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TRANSITION 


ABSTRACT 


This thesis evaluated the Department of Defense's Advanced Concept Technology 
Demonstration (ACTD) process and the challenges encountered in transitioning an 
ACTD to an acquisition program. The methodology included case analysis of thirty- 
eight ACTD program business plans. Nineteen of the programs transitioned while the 
other nineteen were terminated either prior to the Military Unit Assessment (MUA) or 
after. The scope included a review of: 1) ACTD origins and processes as of October 
2007, 2) past ACTD programs, 3) the established documentation criteria associated with 
ACTD selection and evaluation, 4) business plans for the thirty-eight ACTD case 
programs selected for analysis, 5) potential process improvements that would aid in 
ACTD transition to acquisition success. This thesis identified several statistically 
significant variables in the existing ACTD transition process. These variables predict 
transition, or not - they therefore suggest several criteria that should be maintained in the 
ACTD process. Perhaps equally as important, we identified several variables that are not 
significant predictors of transition. Based on our results, we suggest several 
enhancements that could be incorporated into future ACTD processes that may improve 


the insertion of technology to the war fighter. 
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I. INTRODUCTION 


During the 1980s and 1990s, the Department of Defense (DoD) was facing a 
dilemma with regard to its technology management. One key issue was the time between 
project conception and utilization. One solution would find its beginnings in the 
recommendations of the Packard Commission of 1986 [President’s Blue Ribbon 
Commission on Defense Management, Jun 1986] and the Defense Science Board reports 
of 1987, 1990 and 1991 [Reports of the Defense Science Board, 1987, 1990, 1991]. The 
Packard Commission identified a concept that could be broadly defined as a fieldable 
sample to bridge the gap between prototypes and operational units. The commission’s 
view was summed up as “Operational tests should be combined with developmental tests 
of the prototype to uncover operational as well as technical deficiencies before a decision 
is made to proceed with full-scale development [citation].” In this statement (and similar 
complementary conjectures) was the catalyst for Advanced Concept Technology 


Demonstrators (ACTD). To date, 167 programs have been designated as ACTDs. 


This project is broadly defined to analyze the ACTD process and associated issues 
involved within the process. Our focus is on the predictive value of "management plans" 
for ACTDs. We analyzed the management plans for 19 programs that have successfully 
navigated the ACTD process towards an attempted acquisition transition and 19 that 
failed to transition into the acquisition process. We used multivariate regression to 
examine what factors predict a successful transition, or not. In summary we found the 


following variables were significant predictors. 


i Budget matches the schedule 

2. Technology Maturity 

3. Risk Assessment 

After laying out our analysis we discuss our findings and suggest managerial 


recommendations. 


A. BACKGROUND 


As defined in Department of Defense Directive (DoDD) 5000.1 the primary 
objective of Defense acquisition is to acquire quality products that satisfy user needs with 
measurable improvements to mission accomplishment and operational support, in a 
timely manner, and at a fair and reasonable price. [DoDD 5000.1, January 2001] 
Department of Defense Instruction (DoDI) 5000.2 identifies technology transition 
mechanisms designed to ensure the transformation of innovative concepts and superior 
technology to the user and acquisition customer through: 1) Advanced Technology 
Demonstration (ATD) programs, 2) ACTD programs, and 3) Experiments. [DoDI 5000.2, 
January 2001] 


The formal acquisition process, as directed by DoDD 5000.1 and DoDI 5000.2, is 
the primary mechanism for the procurement of new systems and the introduction of new 
capabilities via new or upgraded systems. Recently, it has been recognized that the 
ACTD process, as a pre-acquisition event, provides an important mechanism and 
opportunity for the war fighter to try out and evaluate proposed technology solutions to 
urgent military needs. [ACTD Introduction, September 2001] Each ACTD is aimed at 
one or more war fighting objective and is reviewed by the Services, Defense Agencies 
and the Joint Staff. Key criteria by which ACTD candidates are evaluated consist of: 1) 
Response to user needs, 2) Exploit of mature technologies, and 3) Potential effectiveness. 
[ACTD Guidelines - Introduction to ACTDs, May 2001] An ACTD is designed to 
provide a sound assessment of the military utility of a proposed solution prior to a 
decision on formal development or acquisition. The purchase of additional capability 
beyond the residuals provided by the ACTD, where appropriate, is accomplished through 


a formal acquisition program. 


While identified as tools to rapidly transition technology to the war fighter, it is 
not certain whether ACTDs live up to their expectations. As defined in greater detail later 
in this thesis, ACTDs are two to four year programs that, if successful, may be 
transitioned to the war fighter as residual assets, for two or more years, or as a new 


acquisition program. Utilization of residual assets alone typically lack the logistics chain 
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associated with standard DoD program, thus limiting useful life. The acquisition 
transition process however, currently requires funding, via the Program Objectives 
Memorandum (POM) cycle, along with the appropriate DoD acquisition related 
documentation to be available/completed before the process can move forward. These 
combined events impart a two-year acquisition transition window following the 
successful completion of an ACTD program, which adversely impacts program 
momentum. Additional momentum impacts include changes in user organizations, 


sponsor organizations or lead service organizations. 


The ACTD process has a significant level of management oversight, however 
each program is highly tailored and there is a much less formal structure than with the 
standard acquisition process. The standard process typically involves programs with 
higher funding levels, which are therefore governed by laws and regulations, which have 
to be addressed by major defense acquisition programs. For those ACTDs that 
demonstrate strong military utility, the intent is to transition into the formal acquisition 
process to acquire the system in sufficient quantity to meet the operational requirement. 
However, without careful preparation, the transition may result in the loss of some of the 
benefits of the ACTD. For example, without suitable preparation in areas such as 
contracting, costly delays - including a break in a production line - could occur. Upfront 
planning is crucial to ensuring successful transition of an ACTD to the acquisition 
process. Potential outcomes that could be expected depending on the amount of 


groundwork performed could include: 


1. ACTD does not transition because it is judged to lack military utility. 

2. ACTD does not transition because of poor management (or other 
problems). 

3: ACTD transitions, but has problems (due to poor management, etc). 

4. ACTD transitions with no problems. 


The ACTD process appears to be performing its job well, 43 out of 98 ACTDs 
have successfully completed the demonstration phase based on DoD statistics. However, 
ACTD transition to a DoD 5000 series acquisition project remains a hurdle with only 32 


out of 98 ACTDs currently being executed as acquisition programs. [Joint War fighting 


Science and Technology Plan, February 2002]. Tailoring of this process or defining 
transition needs to benefit the program and the war fighter is required. This thesis will 
attempt to define those elements that have helped or hindered ACTD transitions and 


establish guidelines to assist transitions in the future. 
B. PURPOSE 


The purpose of this project is to help improve ACTD program management by 
identifying attributes which predict transition into the acquisition process. These 
attributes can furthermore be used to identify potential pitfalls in existing ACTD 


programs. 
C, RESEARCH QUESTIONS 


The research questions associated with this project consists of: 
Primary 


1. What are the most important variables that will affect success or failure of 
an ACTD to transition into the normal acquisition process? 


Secondary 


1. Were all managers identified by name? What effect does the inclusion or 
exclusion of managers identified by name have on an ACTD’s success? 


2: Does the funding profile correspond to the development schedule? What 
effect does the budget profile matched to a program’s development 
schedule have on an ACTD’s success? 


3: Was a well-defined military need included in the ACTD’s management 
plan? What effect does the inclusion or exclusion of a thoughtful military 
need have on an ACTD’s success? 


4, How mature was the technology (especially, software technology) noted 
for inclusion in the ACTD program? What degree of commercial/off-the- 
shelf versus new development was utilized? What effect does the maturity 
of a program’s technology have on an ACTD’s success? 


De Was a detailed transition strategy included in the management plan? What 
effect does the inclusion or exclusion of a detailed transition strategy have 
on an ACTD’s success? 


6. Did the program’s development fall within the prescribed development 
timeline of two to four years? What effect does a development schedule 
contained to two to four years have on an ACTD’s success? 


te How in-depth was the program’s management plan? What effect does the 
depth of a management plan have on an ACTD’s success? 

8. How many parties were involved in management oversight of the 
program? Do fewer or more parties have an impact on an ACTD’s 
success? 

9. How much capital investment was required as laid out in the management 


plan? Does the degree of capital requirement have an effect on an 
ACTD’s success? 


10. How complex is the ACTD’s technology? To what degree does the 
interrelationship between technology efforts determine technology 
complexity? What effect does technology complexity have on an ACTD’s 
success? 


Th How risky is an ACTD effort? To what degree do all programmatic 
efforts analyzed holistically have on a program’s risk? What effect does 
assessed risk have on an ACTD’s success? 


D. BENEFITS OF STUDY 


This thesis is intended to define attributes that involves the ability of ACTD 
programs to transition to the acquisition process. Its findings may have value by 


suggestion how ACTD management may be improved. 


E. SCOPE AND METHODOLOGY 


The scope of this thesis is to: (1) review of the ACTD process as defined under 
DoD 5000 documentation, (2) review past ACTD programs, (3) perform an analysis of 
multiple ACTD program management plans, (4) investigate potential areas affecting 
ACTD successes, and (5) define potential pitfall identification which would aid in ACTD 
transition to acquisition programs. This thesis is intended to identify failings in the 
existing ACTD transition process and conclude with recommended improvements that 
enable a more stable and rapid introduction of technology to the war fighter through the 


acquisition process. 


The methodology used in this thesis research consists of coding programmatic 


elements and statistical analysis 


F. ORGANIZATION OF STUDY 


This thesis is organized into four primary sections, the first of which discusses the 
acquisition reform revolution. This section will describe the ACTD process that is in 
place today. The second section will review the transition process and associated risk 
involved with the ACTD process. The content will consist of the transition plan, 
contracting strategy, and the transition funding along with the risk involved. These 
sections will be followed up by an analysis of 38 randomly chosen ACTD programs that 
were measured against the research questions. These 38 programs were run in a multiple 
regression and correlation analysis to determine what variables affect the success or 
failure of an ACTD to transition into the normal acquisition process. We identified 
several statistically significant variables. The final section concludes with 


recommendations about how to improve the ACTD process in the future. 


Ul. THE ACTD PROCESS 


ACTDs were first introduced in 1995 with twelve authorized demonstrations. 
These programs started the revolution towards a more rapid acquisition process that is 
still ongoing today. This process allows the U.S. government to reduce its cycle time and 


speed its delivery of advanced capabilities to the war fighters (Aldridge, April 12, 2002). 


Chapter IT will discuss the evolution of the ACTD process; the initiation, 
developments implemented in the ACTD lifespan, the process and goals associated with 
the ACTDs, and will close with a brief review of the ACTD programs initiated between 
FY95 and FY07. This information will help the reader understand and appreciate how 


ACTD programs are integrated into the acquisition system. 


A. ACTD BEGINNINGS 


In 1986, Advanced Concept Technology Demonstrations were identified by the 
Packard Commission as an idea without a name. They presented this idea of improving 
DoD management and organization in a Presidential report (Packard, 1986). The report 
dealt with improvements to the defense acquisition system and identified several means 
to obtain acquisition reform. Their findings were critical elements for the process 
changes that followed. The commission believed that through the use of demonstration 
platforms, or prototypes, the government could streamline procurement practices to 
reduce costs while at the same time gain a realistic assessment of operational suitability 


(South, 2003). 


Secretary of Defense Dick Cheney took this idea from the Presidential report of 
1986 and defined a new acquisition strategy in his 1992 annual report to the President 
and the Congress (Cheney, 1992). The use of demonstration platforms instead of the 
traditional production programs allowed the military to validate new concepts. This 
would reduce procurement timelines so Cheney recommended that proven subsystems or 


technologies be inserted into existing weapons platforms (Cheney, 1992). 


During that same time Representative Les Aspin grabbed Cheney's idea and 
developed a "Rollover-Plus" plan (Aspin, 1992). 
We would not commit to quantity production at the outset of the 
development. Instead, a prototype would not be brought into full-scale 
production until the resulting component or system met stringent criteria. 
Those criteria are a) the technology works, b) it is required by 
development of the threat, or c) represents a breakthrough that would alter 
battlefield operations. If the resulting prototype did not meet those 
criteria, however, we would "rollover" the new technologies and lessons 


learned from development into a further iteration of engineering, 
development, and prototyping. (Aspin, 1992). 


Secretary Cheney finally gave this concept a name in his 1993 annual report. He 
came up with guidelines for Advanced Technology Demonstration (ATD) that allowed 
an ATD to transition into production (Cheney, 1993). His approach on the ATD is that it 
could not stand alone. It will need exercises and simulations to prove the technology is 
ready, manufacturing process is available, and operations are understood before the ATD 
is considered. Each ATD is required to demonstrate to decision makers that the 
technology is feasible, affordable, and compatible with operational concepts (Cheney, 
1993). The intent of the process was to provide realistic demonstrations of the 
technology development and involve the war fighter in the evaluation process prior to 


commitment of funds (Cheney, 1993). 


In January of 1994, Les Aspin took over as Secretary of Defense. In his first 
annual report to the President he referred to the ATD as it became known from Secretary 
Cheney to Advanced Concept Technology Demonstration (ACTD). Secretary Aspin 
went on to say that ACTD's need to have an integrated effort between operational users 
and Science and Technology (S&T) community. Where the user provide the operational 
context, concept of operations, and manages the operational aspects of the demonstration; 
while the S&T community provides the advanced technology elements (Aspin, 1994). 
The point of this is to address operational utility and have a cost effective program with 
minimal risk involved. By refining the operational requirements and concept design will 
allow the new capability to enter the formal acquisition process with minimal delay and 


cost (Aspin, 1994). 


Aspins ACTD approach emphasized cooperation between the war fighter and the 
S&T communities. The ACTD concept would provide the traditional role of technical 
and cost risk reduction, while also providing a way for refining the operational concept. 
Concurrent with these efforts, Aspin created the position of Deputy Under Secretary of 
Defense (Advanced Technology) (DUSD(AT)) to effectively manage the ACTDs (South, 
2003). 


With the base line for ACTDs in place, Secretary of Defense William Perry 
continued to add to the process. His change allowed the war fighter to modify the ACTD 
as it evolved through the process of fielding and operational testing (Deutch, 1994). An 
important aspect for the user of the ACTD is that it has operational capability for 
continued use. This allows the combat commander flexibility to refine the doctrine and 


tactics to maximize the technologies capabilities. (Deutch, 1994). 


Perry continued to improve the ACTDs and made them official in his 1995 annual 
report. Most of the earlier ideas for ACTDs were still in place including early 
involvement by users, refinement of operational concepts, fielding, and quick transition 
to the field. In his annual report he also introduced the following four criteria for an 
ACTD to be considered (Perry, 1995). 


1. Offers a potential solution to a military problem or introduces a significant 
new capability. The Joint Requirements Oversight Council (JROC) and 
the unified commanders must approve and ACTD. 


2 Is relatively mature and contributes to solving the problem. 
Has and executable program and management plan. 

4, Is a two to four year program that can be supported for two years in the 

field? 

The 1995 annual report also defined outcomes for ACTDs. If unsuccessful, 
ACTDs were to be terminated or shelved for future restructuring. Upon the user's 
recommendation, an ACTD could be directly fielded with minor modifications or enter 
the formal acquisition process at and advanced milestone (MS B or MS C) (South, 2003). 
Although he did make it clear that the ACTD process was not a substitute for the formal 
acquisition process and was not to be used as a vehicle to purchase large, complex 


weapon systems such as ships and tanks. It was also not intended to support acquisitions 
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of new systems such as vehicles or munitions. With these restrictions Secretary Perry 
prevented the ACTD programs from directly fielding expensive defense acquisition 


programs. 


In 1995, the first twelve ACTD programs were selected. Because these programs 
are not followed with the same oversight as typical acquisition programs they have 
collected the interest of political officials. In 1997, the Office of the Inspector General 
(OIG) audited the ACTD process (South, 2003). The major topics the OIG wanted to 
look at were 1) the criteria used to select current and pending ACTD efforts, 2) the 
process for determining the program's effectiveness, and 3) the transition of the program 
into the defense acquisition cycle. They also evaluated 4) the adequacy of the DoD 
management controls as they applied to the audit objective (Office of the Inspector 
General, 1997). OIG investigated 9 of the 22 ACTDs approved in FY95 and FY96. 
They found five were questionable choices based on their interpretation of the ACTD 


selection criteria (South, 2003). 


OIG also found four projects did not have mature technology. All four ACTDs 
relied significantly on modeling and simulations because supporting programs were not 
mature. A recommendation of a clear and consistent criteria selection for mature 
technologies to be established (South, 2003). They also reported that eight of the nine 
programs assessed did not have a declared urgent military need. OIG recognized that 
military need may be declared by certain DoD officials but indicated that DUSD(AT) had 
not defined what constitutes an urgent military need or who may declare the urgent need 
for the ACTD candidates (South, 2003). OIG recommended that a critical military need 
be defined. 


In 1998, General Accounting Office (GAO) received a request from the House of 
Representatives to take a closer look at the ACTD process. Specifically they wanted 
GAO to determine: 


1. Whether the selection process included criteria that were adequate to 
ensure that only mature technologies were selected for ACTD prototypes. 


2: Whether guidance on transitioning to the normal acquisition process 


ensured that prototypes appropriately completed product and concept 
development and testing before entering production. 
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3. 


Whether DoD was procuring more ACTD prototypes than needed to 
assess the military utility of mature technology (Rodrigues, 1998). 


What GAO found was a great need for improvement within the ACTD process. 


GAO reported back to the House of Representatives that: 


h. 


DoD's process for selecting ACTD candidates did not include adequate 
criteria for assessing the maturity of the proposed technology resulting in 
the approval of ACTD projects that included immature technology. 


Guidance on entering technologies into the normal acquisition process was 
not sufficient to ensure that prototype completed product and concept 
development and testing before entering production. 


DoD's practice of procuring prototypes beyond those needed for the basic 
ACTD demonstration and before completing product and concept 
development and testing was unnecessarily risky (Rodrigues, 1998). 


With all these findings the GAO stated three recommendations for the Secretary 


of Defense to take action on in order to clarify the ACTD process: 


1. 
Di 


Ensure the use of mature technology with few, if any, exceptions 


Describe when transition to the development phase of the acquisition cycle 
is necessary and the types of development activity that may be 
appropriate. 


Limit the number of prototypes to be procured to the quantities needed for 
early user demonstrations of mature technology until the items product 
and concept development and testing has been completed (Rodrigues, 
1998). 


One way to approach the maturity level issue would be to establish a criteria 


based of the DoD 5000.2-R manual that list various Technology Readiness Levels (TRL) 


which includes a description of what each of the levels mean including examples. In 


2002 a gentleman named Jim Sheldon expanded the chart to include component/system 


levels, equipment necessary to demonstrate capability and potential operation 


environment. Based on the TRL chart, level 1 through 4 would not be acceptable levels 


for ACTDs. 


TRLs 5 and 6 could possibly be acceptable. TRLs 7 through 9 would 


appear to directly satisfy the intentions of the ACTD maturity level (South, 2003). 
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Technology 


Readiness 
Levels 


3) Analytical amd 
experimental 
critical function 
and/or 


characteristic 
proof of concept 


4) Component and’ 
or breadboard 
validation in lab 
environment, 


PPT Tew isla (iyi) 


Lowest level of technology readiness. 
Scientific research begins to be 
translated ito applied research and 
development. Examples might 


principles are observed, ‘practical 
applications can be invented, The 
application 1s speculative and there as 
no proof or detailed analysts to 
support the assumption. Examples are 
still limuted to paper stuchies. 

Active research and developmyent is 
initiated. This includes analytical 
studies and laboratory studies to 
physically validate analytical 
predictions of separate elements of 
the technology. Examples include 
components that are not yet integrated 


integrated to establish that the pieces 
will work together. This is relatively 
“low fidelity” compared to the 
eventual system. Examples include 
integration of “ad hoc” lardware in a 
laboratory. 
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HW/SW 
tet tbe ch 
Demonstrate 


Capability 


Level 


Low oe 


(integration Beers nonscale 


Environment 





Technology 
Readiness 
Levels 


6) System subsystem 
model or 
prototype 
demonstration in 
relevant 
enviromment 


7) System prototype 
demonstration in 
an operational 


enviromment 


$8) Actual system 
completed and 
flight “qualified” 
throush test and 


demonstration 


9) Actual systean 
“flight proven” 
through successful 
mission operations 


PTStery telat 


Representatrve model or prototype 
system: which is well beyond the 
breadboard tested for technology 
readiness lewel (TRL) 5_ is testedim a 
relevant environment, Represents a 
mngjor step up ina technology's 
demonstrated readiness. Examples 
usclude testing 2 prototype in a high 
Sdelity laboratory environment or in 
x sinwilated 

Prototepe near or at plammed 
operational systems, Represents a 
muzjor step wp from TRL 6. reoumnz 
the demonstration of an actual system 
Prototype in an operational 
enviromment, such 25 in an aircrait. 
vehicle or space. Examples include 
testing the prototype in 2 test bed 
adrcrait 

Technology has been proven to work 
tn its final fom aad under expected 
conditions. In almost all cases, this 
TRL represents the end of nue system 
development, Example: include 
developmental test and evaluation of 
the system in its Intended weapon 
Syttenz to determine if it meets design 


Actual application of the technology 
mm its final form and under mission 
conditions, such a3 those encountered 
mm operational test and evaluation In 
almost all cases, this is the end of the 
Last “bug fixing” aspects of tue 
system development. Framples 
inchide wsing the system under 
operational mission conditions 


HW/SW 


Cree ame th 


Environment 


Demonstrate 
Capability 
Prototype Lab or konited 
(should be -very close demonstation 
to fomn, fit and 
function) 


Prototype 
(form, &t and Ametion) 


Actual system in final OTE in oparational 
form mussion conditions 





Table 1. | Technology Readiness Level (From: Sheldon, August 2002) 


Since the beginning ACTDs have been scrutinized heavily which has led to 
multiple improvements. For example, transition manager were originally defined at the 
completion of the ACTD program. Currently with the more proactive execution attitude 
all ACTD programs must have an established transition manager before being considered 


for execution approval. 
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The ACTD process evolved in 1994 in response to the recommendations of the 
Parkard Commission of 1986 and the Defense Science Boards of 1987, 1990, and 1991. 
As can be seen it has been through many variations and continues to change as the global 
environment changes. While these changes have affected the process they have not 
effected its execution (South, 2003). Since its inception, a total of 167 ACTDs have been 
initiated from fiscal years 1995 through 2007. 


B. ACTD PROCESS 


Advanced Concept Technology Demonstrations (ACTD) is a complicated 
program that many customers in the defense industry do not understand. ACTDs are 
capabilities demonstration and evaluation programs in which the development and 
employment of technology and innovative operational concepts by the military user are 
the primary focus (Defense Acquisition University, 2006). The definition still does not 


explain much. 


The ACTD process is a pre-acquisition activity with a significant level of 
management oversight, but each program is tailored and a much less formal structure 
than the actual acquisition process (Office of the Deputy Under Secretary of Defense for 
Advanced Systems & Concepts, August 2004b). This typically causes problems when 
transitioning from an ACTD to the acquisition process because it is not governed by laws 
and regulations the same as the normal acquisition process. For this reason some lessons 
learned from past transition problems are addressed in the most current version of the 


Department of Defense (DoD) 5000.2 document. 
1. Objective 


An Advanced Concept Technology Demonstration (ACTD) is a joint effort by the 
acquisition and operational communities within the DoD. Typically, ACTDs begin by 
identifying significant military needs, and then matching them with technology programs 
ready to focus on a military application. The emphasis in ACTDs is a current fix in order 
to validate joint military needs. These fixes are typically technology based and usually 


include new operational concepts and new organizational structure. The fixes must be 
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affordable, interchangeable, sustainable, and capable of being expanded as the 
technologies and threats change (Office of the Deputy Under Secretary of Defense for 
Advanced Systems & Concepts, August 2004a). The acquisition approach over time is an 


important part of the ACTD concept. 


The initial requirements and the initial design reflect the performance achievable 
with current technology, but provisions are included to encourage growth. The basic form 
of an ACTD generally starts from a collection of mature technologies or technology 
demonstration programs which are key technologies. The technologies are combined and 
integrated into a complete military capability. The objective is to provide decision makers 
an opportunity to fully understand the operational potential offered by a proposed new 
military capability before making an acquisition or long term decision. This objective is 
met by developing useful prototypes of the proposed capability and providing those 
prototypes to the war fighter for evaluation (Office of the Deputy Under Secretary of 
Defense for Advanced Systems & Concepts, August 2004a). The war fighter first 
develops operational concepts designed to fully challenge the proposed capability, and 
then uses the prototypes and associated operational concepts in realistic military exercises 
to assess the resulting military utility. During the assessment of the ACTD, the user also 
determines the broad statement of need. This was first introduced in the proposal of the 
ACTD, and should have a definite set of operational requirements that can support a 
follow-on acquisition (Office of the Deputy Under Secretary of Defense for Advanced 
Systems & Concepts, August 2004a). At the completion of an ACTD, the remaining 
systems used in the evaluation process are left with the user to provide a temporary 


capability or in some cases to fulfill the total current need. 


Each ACTD should provide a serious need for new or increased military 
capability. This need is usually provided by the operational war fighting community 
(JCS, CINCs, Service operational organizations). Although some ACTDs focus on a 
service specific capability, the highest priority in the selection process is placed on joint 
capabilities. ACTDs have become an important vehicle for addressing joint needs (Office 
of the Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 
2004a). 
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A primary ground rule for any ACTD is the active participation of a sponsor or 
user organization, in partnership with a service which will serve as the Technical 
Management Office. A proposal for an ACTD must identify and develop this user and 
developer partnership before consideration can be given. The initiation is either by the 
acquisition community, or by the war fighter community. The interests of the war fighter 


are very important and the guidelines regarding ACTDs are considered flexible. 


The Deputy Under Secretary of Defense (Advanced Technology) (DUSD/ 
AS&C) has the oversight responsibility for the ACTD program. He is responsible for 
developing and issuing guidance regarding the ACTD program, for evaluating candidates 
and approving new ACTDs. He is also going to provide oversight, support and 
evaluation of ongoing ACTDs (Defense Acquisition University, 2006. This document 
describes the process for formulating, and evaluating ACTD candidates, and for 


approving and initiating ACTDs. 


The goals of the ACTD processes are to accelerate and facilitate responses to 
priority military needs with a combination of new and fielded hardware and/or software, 
confirming that transformational technology is appropriate for military use, develop 
CONOPS trough resources, and creating an organizational structure that satisfies those 


needs (South, 2003). 


To satisfy the objectives guidelines have been developed which apply to both the 
proposed capability and to the program for developing and evaluating the capability. 
These guidelines have been referred to as the ACTD selection criteria, they are intended 
to provide guidance for formulation of candidates, as well as structure during the ACTD 
process. The criteria are as follows: (Office of the Deputy Under Secretary of Defense 
for Advanced Systems & Concepts, August 2004a). 


1. The timeframe for completing the evaluation of military utility is 2-4 
years. 


The technology should be sufficiently mature. 
Provide a potentially effective response to a priority military need. 


Lead service/agency has been designated. 


fe ee ee tO 


Risks have been identified. 
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6. Demonstrations or exercises have been identified. 
Te Funding is sufficient. 


8. Developer is ready to prepare a plan that covers all essential aspects. 


There are other factors that should be considered during the ACTD formulation 
phase. Although these are not selection criteria per se, they do lead to better alignment 
with the objectives of the ACTD program and may affect the level of support a given 
candidate receives during the selection process. As indicated earlier, the emphasis in 
ACTDs is on near-term responses to the need, responses that are affordable, 
interoperable, sustainable, and capable of being evolved as the technology and threat 
change (Office of the Deputy Under Secretary of Defense for Advanced Systems & 
Concepts, August 2004a). 


2. Selection Process 


The ACTD process is marked by three basic phases: selection of the projects, 
demonstration of the technologies, and residual use of prototypes or the transition to 
acquisition programs (United States General Accounting Office, 2002). The selection 
process begins with the submission of proposed Advanced Capability Technology 
Demonstration in response to the data call issued by the Deputy Under Secretary of 
Defense (Advanced Technology) (DUSD(AS&C) in October of each year. When 
submitting a particular technology/concept as an ACTD, a one to three page description 


of the proposal should be provided and should include the following information: 


Describe the perceived military need, urgency of timing, and potential utility of 


the candidate system. 
Paragraph(s) describing the basic technology/concept. 
Paragraph describing the type of demonstration envisioned. 


Participants in the ACTD. To what degree will the proposed ACTD support 


joint/combined operations? 


Overall funding required, proposed funding sources and the schedule for the 
ACTD. 
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Paragraph describing perceived technical, funding, and schedule risks of the 


proposal. 


Is the ACTD primarily directed towards Dominant Maneuver, Precision 


Engagement, Full-Dimensional Protection, or Focused Logistics? 
Proposed Lead Service/Agency and User-Sponsor. 
Briefly describe envisioned residual assets and ACTD transition strategy. 


Point(s) of contact for the ACTD submission. Include name, rank, organization, 
phone number, fax number, and addresses (mail / e-mail). (Office of the Deputy Under 


Secretary of Defense for Advanced Systems & Concepts, August 2004a). 


Once a developer and user team has submitted the information described above, 
the candidate review process begins. A specific individual within the ODUSD(AS&C) is 
designated to serve as the point-of-contact (POC) for each ACTD candidate. That person 
will work with the individual organization to ensure the proposal is complete and 
coordinate the review process (Office of the Deputy Under Secretary of Defense for 
Advanced Systems & Concepts, August 2004a). A typical schedule for this process is 


shown in the Figure 1 below. 


The first step in the process is to prepare a briefing package (no longer than 30 
minutes) to be presented first to the DUSD(AS&C) due the January after the data call is 
pronounced. The briefing should define the top level mission need, the ACTD objectives, 
operational concept, technical approach, time scale, anticipated program cost (with 
funding sources identified and/or required, including out-year, OSD ACTD funds), the 
management structure, the primary participants, the anticipated capability and, should 
address the "ACTD Selection Criteria.” (Office of the Deputy Under Secretary of 
Defense for Advanced Systems & Concepts, August 2004a). Doing this first step 
correctly is the key to success. If this step is not complete then the whole proposal is at 


risk. The first impression is what makes or breaks the success of the proposal. 
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The briefing should be presented by both the user and developer teammates. The 
user outlines the mission need and operational concept, and the developer presents the 
technical concept and programmatic approach, highlighting a clear statement of the end 
product (the interim capability). The USD(AS&C) staff stands ready to assist in the 
development of this briefing, as well as the scheduling, as necessary (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 2004a). 
With the USD staffs assistance this initial submission should be perfect. 


The result of this initial briefing will be one of the following courses of action: (a) 
acceptance of the ACTD as a formal "candidate" and scheduling for presentation to the 
full AT/BC principles for discussion and recommendation; (b) critique by DUSD(AS&C) 
and request for revision and follow-up presentation; or (c) determination that the concept, 
for whatever reason, is outside the scope of the ACTD process (Office of the Deputy 
Under Secretary of Defense for Advanced Systems & Concepts, August 2004a). 


After the DUSD(AS&C) briefing the proposal goes through a couple initial 
reviews (Advanced Technology Breakfast Club (AT/BC) initial review from February to 
March, then a status review with the hill in April) before being submitted to the Breakfast 
Club (BC) or a detailed review. The BC will meet at the beginning of April and usually 
finish up by the beginning of June. BC is comprised of the senior Science & Technology 
(S&T) representatives from the services, Director of Defense Research & Engineering 
(DDR&E), Command, Control, Communication & Intelligence (C31), and selected 
Department of Defense (DoD) agencies, representatives from the Operational 
Departments of the Services and from the Joint Staff (Defense Acquisition University, 
2006). After the BC is done with the proposal it is then submitted to the Joint Staff for 


another review that takes place from June to August. 


The DUSD(AS&C) will consider the recommendations of the AT/BC and the 
Joint Staff and make the decision to retain the specific ACTD candidate for presentation 
to the Joint Requirement Oversight Counsel (JROC). The JROC reviews and 
recommends prioritization of ACTD candidates based on military need. At this point, 
information on the candidates is provided to the Congressional Authorization and 


Appropriations Committees to support their committee marks. A final review, termed the 
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‘Final Scrub', is then conducted prior to the start of the fiscal year (Defense Acquisition 
University, 2006). Of the candidates selected by the Joint Staff and OSD are most 
deserving of ACTD status. The focus of this review is once again on the selection 


criteria, with the addition of two other topics; transition strategy and proposed ACTD. 


The ACTD list is then coordinated with the Vice Chairman, Joint Chief of Staff 
(JCS) and the Under Secretary of Defense (Acquisition and Technology) before the final 
ACTD Implementation Directives (ID) for the approved ACTDs are signed by the 
DUSD(AS&C) (Aldridge, April 12, 2002). 


ACTD FY-2007 ACTD/JCTD Program Timeline 


Selection Process for the FY 2007 ACTD/JCTD Candidates 
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Figure 1. Program Timeline (From: Carson PPT, 2006) 


At the same time the briefing is presented to the BC, a draft Implementation 
Directive should be in work and presented to AS&C staff once the candidate is selected. 
The final selection of ACTDs will not occur until the Defense Appropriations Bill has 


been signed. However, soon after that time the final ACTD approvals can be granted. 
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Approvals are in the form of the DUSD/AS&C signature on the Implementation 
Directives, so it is imperative the staffing of the Implementation Directives be completed 
in a timely fashion (Defense Acquisition University, 2006). Even though there is not a 
set timeline on the ID, it is the highest priority once the selections occur. A late 


document will risk no final signature and a cancelled ACTD. 


The final step of ACTD process is the completion of the ACTD Management Plan 
(MP). The items that are addressed in the management plan include the following: 
(Office of the Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004a). 

1. The objectives that the ACTD must demonstrate. 

2 The overall approach. 

i The concept and technical approach of the ACTD 
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The programmatic and organizational approach, which includes the key 
decision makers. 


3: The approval agencies 
The endorsements of the ACTD participants. 
de Any modifications associated with the ACTD 


The process of working out the details of the Management Plan to the satisfaction 
of all involved will take some time and this process is viewed as a very productive 
element of the ACTD (Office of the Deputy Under Secretary of Defense for Advanced 
Systems & Concepts, August 2004a). It is necessary for the Plan to receive full 


endorsement within 90 days or less after approval of the Implementation Directive. 


For this project, the most important section of the management plan is the 
programmatic and organizational approach, item 4, which include key players and the 


transition plan. 


The initial meeting of the ACTD Oversight Group should be scheduled during the 
first year of the program to confirm the planned direction of the program and if necessary 
to resolve any outstanding issues relating to the Management Plan (Office of the Deputy 
Under Secretary of Defense for Advanced Systems & Concepts, August 2004a). The 


Management Plan is intended to provide a baseline program definition, as well as, a 
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practical and flexible learning environment in which operations and technical concepts 
can be traded off and refined prior to entering the formal acquisition process. Only 
significant modifications to the Plan need be approved by the ACTD's Oversight Group 
(Office of the Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004a). 


3. ACTD vs. Acquisition Funding 


ACTD program managers must obtain ACTD and any follow-on acquisition 
funding through the Planning Program and Budgeting System (PPBS) just like traditional 
acquisition program managers. While traditional acquisition programs should be fully 
funded in the Future Years Defense Plan (FYDP), ACTD programs are not required to 
include funding for post-ACTD activity in the FYDP. At first glance this benefit of not 
funding additional research and development (R&D) or any production effort may appeal 
to the Services and OSD in a fiscally constrained environment; however, it is not 


practical and creates problems as ACTDs transition to acquisition programs (Mol, 1998). 


In reality, post-ACTD financial planning must be accomplished during the ACTD 
since the acquisition Milestone Decision Authority (MDA) will only transition the 
program from an ACTD to an acquisition program if the follow-on effort is fully funded. 
This problem is recognized within the acquisition community since it affects not only the 
ACTD and its follow-on acquisition effort, but also other modernization programs 
competing for the same scarce funding (Mol, 1998). As mentioned earlier if the initial 
proposal is not complete on time the ACTD could risk failure due to a lack of funds. 


This is why that first step in the selection process is so critical to the existence of ACTDs. 


OSDs ACTD guidelines offer three strategies to deal with this problem if the 
funding was not properly planned for. First, the services can appeal directly to OSDs 
Defense Resources Board (DRB) to include funding for the follow-on acquisition effort. 
If this brute force method is successful, it means OSD will transfer funding from an 
approved program to the new ACTD follow-on effort. This method disrupts the PPBS 


process by placing new funding requirements very late in the process after priority and 
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funding issues should have previously been resolved within the services (Mol, 1998). 
This method is not very popular and most likely will not happen unless it comes from 


very high in the chain-of-command. 


The second alternative suggests the acquisition strategy contain a two-year gap 
between the completed ACTD and the beginning of the formal acquisition process. This 
gap allows program managers time to obtain funding through the normal two-year POM 
process. While this suggestion creates efficiency within the PPBS process, it is likely to 
break the program and cause its cancellation due to the increased contractor shut down 
and startup costs (Mol, 1998). Historically this has been a major cause of ACTDs to fail. 
As we will see later there has been a patch to fix this problem by using OSD funds from 
the RTD&E funds budgeted for this purpose. With that being said it still needs to be 


planned for in the initial proposal. 


The third, and probably most attractive solution offered, is to assume success. If 
the acquisition strategy includes this course of action, the services must insert an 
acquisition cost estimate into the PPBS process before the ACTD testing is complete and 
before the user has had an opportunity to make an operational assessment (Mol, 1998). 
Unfortunately, not having the test results will build uncertainty into the cost estimate and 
increase the funding since results obtained in the last year or two of the ACTD are the 
most important. During this critical time DOD will determine the production 
configuration, the type of funding required (R&D vs. Production), and the scope of any 
future effort (Mol, 1998). Consequently, the Services may be reluctant to fund any 
follow-on effort given the ACTD’s unpredictable future. 


A specific example would be programming funds in the POM cycle for follow-on 
production of an ACTD where success is anticipated, such as for Global Hawk, even 
though flight testing has not yet demonstrated high military utility (Mol, 1998). The 
Army already has a similar strategy in place to fund emerging technologies, such as 
Advanced Technology Demonstrations (ATDs) and Advanced War fighting Experiments 
(AWEs). During the development process of the FY98-03 POM, the Army established a 
Task Force 21 budget line, with RDT&E funds identified and submitted in the FY98 


budget request. The establishment of the RDT&E line, to support Force 21 requirements, 
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provides the service the flexibility to leverage, exploit and transition new technologies, 


buy prototype systems, and put them in the hands of the soldiers quickly (Mol, 1998). 


In the end, the funding rules are different between the two systems. PPBS reality 
dictates ACTD programs and it must have a plan and program for follow-on acquisition 
funding in the FYDP to maintain program stability. ACTD program managers and the 
service headquarters must incorporate their budgets into the PPBS—yust like traditional 


acquisition programs. 


4. Current Funding Process 


Programmatic flexibility and speed in adjusting to change are critically important 
to success with a program as technologically intensive as the ACTD. In the current 
environment, technology is accelerating at a tremendous rate. Our speed and flexibility to 
leverage, exploit, and transition mature or emerging technologies into the operational 
force structure is hampered by resource and budget constraints (e.g., the inability to 
perform timely programming of funding during the Program Objective Memorandum 
(POM) process) (Office of the Deputy Under Secretary of Defense for Advanced Systems 
& Concepts, August 2004b). If the selection process is used correctly it can minimize 
some of these resource and budgeting issues that become major road blocks for future 


requirements. 


At the time a proposed ACTD is approved, the Deputy Under Secretary of 
Defense (Advanced Technology) also approves the funding for an ACTD, to include any 
supplemental funding provided by OSD. The Executing Agent will designate an ACTD 
Technical Manager (TM) who is responsible for managing the execution of all funds 
associated with an ACTD (Mol, 1998). It is also the responsibility of the TM to develop a 
life cycle cost estimate for the system to serve as a basis for planning, programming, and 


budgeting of the resources by the Lead Service for subsequent acquisition. 


Funding for ACTDs can currently be planned, programmed, and budgeted 
through two sources: 1) The Military Departments or Agencies supplying the underlying 
technologies provide the funding associated with those technology programs, and 2) OSD 


can supplement the service or agency funding to cover cost in three areas: a) added costs 
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incurred when other technology program funds are redirected to support the ACTD; b) 
costs due to any requirement to provide additional quantities of hardware; and c) cost for 
technical support for two years of field operations following the ACTD (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 2004b). 
Although OSD provides start up funding for ACTDs, the military services and defense 
agencies are ultimately responsible for financing the acquisition and support of 
equipment for the ACTD (United States General Accounting Office, 2002). However, 
funding to support the follow-on activity (development, full rate production, or purchase 
of additional quantities of commercial items) is not typically funded in OSD or the 
Service/Agency until the ACTD demonstrates the military utility of the capability being 
assessed (Office of the Deputy Under Secretary of Defense for Advanced Systems & 
Concepts, August 2004b). This lack of prior funding creates a significant challenge that 


must be addressed as part of the transition effort. 


Proper funding of ACTDs is a critical component for successful program 
execution. Funds for ACTDs are submitted in the RDT&E Presidential Budget Request. 
A majority of these resources are pulled from the BA-3 Advanced Technology 
Development funds. Some of the budgeting requirements include funding that must be 
sufficient to complete the planned assessment of utility. Budget request must be 
developed and submitted as a part of the proposed ACTD. The budget must identify all 
costs associated with the design and development of the prototype system, all additional 
units required in the ACTD, all exercises that must be paid by the project, and test 
support costs including any modeling simulation and analysis needed to support the 
utility assessment (Office of the Deputy Under Secretary of Defense for Advanced 
Systems & Concepts, August 2004a). The budget must also include costs related to 
planning and preparations for transition into acquisition, as well as the cost to provide 
technical support for the first two years of fielding the residuals. The lead service is 
assumed to budget for all support costs beyond that point (Defense Acquisition 


University, 2006). 
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Both the Implementation Directive (ID) and Management Plan (MP) require a 
“break-out” of resources. Historically, OSD has provided 15-20% of the total funding 
while the services involved provide the rest. It is the goal of AS&C to provide 
approximately 20% of the total funding and no more than 30% of the actual cash funding 
for any particular ACTD (not including coalition partners) (Peterson, 2005). In addition, 
properly spreading the resources is a critical element of the program. AS&C resources 
should be spread across the life of the project, with no more than half of its funding 
requested in the first two years. At the discretion of the DUSD (AS&C) these metrics 
may be waived, but it should be the exception and considered only on a case by case 
basis (Peterson, 2005). Proposals for OSD funding should be coordinated with 
ODUSD(AS&C) during the formulation phase. Figure 2 shows the current ACTD 


funding model in place today and highlights the challenges involved in the process. 


Current ACTD Funding Model 








Challenges: Goal is to initiate ACTDs within months of a JROC approval. However, two 
year PPBE process creates Service challenge in funding new ACTDs. 
(Difficult to achieve before the third year) 
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Unfunded Requirements (UFRs) during 
execution cause significant risk and 
disruption as OSD tries to “share” the 
UFRs with stakeholders. 









CoCOM Non-DoD Coalition 
Even successful demonstrations risk Support Agencies Partners 
waiting 2 years (or more) for resources to 
be programmed via rigid PPBE process. 
Figure 2. Current Funding (From: Carson PPT, 2006) 
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5. Conclusion 


Although this system has its problems in the end it has been a relatively 
successful program for the DoD. They claim roughly 71% of the ACTDs transition at 
least one product to a program of record or directly to the war fighter. DoD would like to 
see that number climb to over 80% (TechLink, 2005). There has been a considerable 
amount of effort put into shrinking the time it takes to get a product through the 
acquisition process. This program was a way to get high priority items in the hands of 
the war fighter on limited resources. After looking at the process the main problem is the 


budget because it is not in line with the typical PPBE process. 


Figure 3 shows how serious Congress has started to take this program. Prior to 
2001 the appropriations were much lower than what was requested. Since then, Congress 
has issued appropriations above what was requested. This is a good sign for future 
programs brought into the ACTD process. Improvements will continue to be made as 


time goes on with the emphasis on transitioning the ACTDs into the acquisition process. 


ACTD Historical Funding Profile 
Budget Vs. Appropriation 











250 @ Budget Request 








224 @ Appropriation 











200 





50 4 





FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06 








| Starting in FY 2001 Congress fully supports ACTD Budget Request | 
eee el 


Figure 3. Historical Funding (From: Carson PPT, 2006) 
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C. REVIEW OF PROGRAMS 


ACTDs represent a bold departure from the traditional research and development 
acquisition cycle that can take 15 years (Kaminski, March 1997) to field a new weapon 
system. ACTDs typically have a two to four year life span as standalone demonstration 
activities (South thesis). Since the beginning of the ACTD process there have been 167 
programs approved through FY07. Of these, 93 have successfully completed the 
demonstration phases and 55 are still in process. Of those 167 programs 16 have been 
terminated due to lack of military utility or immaturity while three have been place on 
hold. Seventy-four have been placed in the "transitioned on record" category indicating a 


successful transition. ACTDs initiated to date are presented in Table 2 (Carson, 2007). 
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FY 95 ACTD 


Advanced Joint Planning 


Precision SIGINT 
Targeting System 


Synthetic Theater of War 


Low-Life-Cycle-Cost 
Medium-Lift Helicopter 


Kinetic Energy Boost- 
Phase Intercept 
Medium-Altitude 
Endurance UAV 
(Predator) 


High-Altitude Endurance 
UAV 


Cruise Missile Defense 
Phase I 


Precision/Rapid Counter- 
MRL 


Joint Countermine 


Rapid Force Prjection 
Initiative 


Total 11 


FY96 ACTD 


Battle Field Awareness 
and Data Dissemination 


Semi-Automated IMINT 
Processing 


Joint Logistics 


Counter Sniper 


Miniature Air-Launched 
Decoy 


Combat Vehicle 
Survivability 
Navigation Warfare 
Tactical High-Energy 
Laser 

Tactical UAV 

Air Base/Port Bilogical 
Detection 


Combat Identification 


Counterproliferation I 


Total 12 


FY97ACTD 


Integrated Collection 
Management 


Information Operations 
Planning Tool 


Consequence 
Management 


Joint Advanced Health 
and Usage Monitoring 
System 


Rapid Terrain 
Visualization 


Chemical Add-On to Air 
Base/Port Biological 


Detection 


Military Operations in 
Urban Terrain 


Extending the Littoral 
Battlespace 


Counterproliferation II 


Total 9 


FY 98 ACTD 


Migration Defense 
Intelligence Threat Data 
System 


Joint Continuous Strike 
Environment 


Adaptive Course of 
Action 


C4I for Coalition 
Warfare 


Space-Based Space 
Surveillance Operations 


Information Assurance: 
Automated Intrusion 


Detection Evnironment 


Theater Precision Strike 
Operations 


Unattended Ground 
Sensors 


Precision Targeting 
Identification 


Joint Modular Lighter 
System 


Line-of-Sight Anti-Tank 
Joint Biological Remote 
Early Warning System 


Link-16 


Total 13 


FY99 ACTD 


Battle Damage 
Assessment in the Joint 
Targeting Toolbox 
Coherent Analytical 
Computing Environment 
Common spectral 
MASINT Exploitation 
Compact Environmental 


Anomaly Sensor II 


Force Medical 
Protection/Dosimeter 


Human Intelligence 
Support Tools 


Joint Medical 
Operations/Telemedicine 


Joint Thheater Logistics 


Personnel Recovery 
Mission Software 


Small Unit Logistics 


Theater Air & Missile 
Defense Interoperability 


Total 11 


FY00 ACTD 


CINC 21 


Coalition Aerial 
Surveillance & 
Reconnaissance 


Comm/Nav Outage 
Forecast System 


Computerized 
Operational MASINT 
Weather 
Content-Based Info 
Security 

Global Monitoring of 
Space ISR Systems 


Ground-to-Air Passive 
Survillance 


Joint Intelligence, 
Surveillance & 


Reconnaissance 
Multiple Link Antenna 
System 

Quick Bolt 


Restoration of Operations 


Tri-Band Antenna Signal 
Combiner 


Total 12 


FY01 ACTD 


Active Network Intrusion 
Defense 


Adaptive Batlespace 
Awareness 


Advanced Tactical Laser 


Advanced Technology 
Ordnance Surveillance 


Area Cruise Missile 
Defense 


Coalition Combat ID 
Coalition Theater 
Logistics 

Coastal Area Protection 
System 

Hunter Standoff Killer 
Team 

Joint Area Clearance 
Loitering Electronic 
Warfare Killer 
Network-Centric 
Collaborative Targeting 
(formally NCCIS&R) 
Personnel Recovery 
Extraction Survivability 


aided by Smart Sensors 


Tactical Missile System 
Penetrator 


Theater Integrated 
Planning System 


Total 15 


FY02 ACTD 


Active Denial System 


Advanced Notice 


Agile Transportation 


Coalition Information 
Assurance Common 
Operational Picture 
Contamination 
Avoidance at Seaports of 
Debarkation 


Expendable Unmanned 
Aerial Vehicle 


Homeland Security 
Command and Control 


Hyperspectral Collection 
and Analysis 
Joint Explosive Ordnance 


Disposal 


Language and Speech 
Exploitation Resource 


Micro Air Vehicle 


Pathfinder 


SIGINT Processing 


Space-Based MTI 


Thermobarics 


Total 15 


FY03 ACTD 


Adaptive Joint C4ISR 
Node 


Counter Bomb/Counter 
Bomber 


Deployable Cargo 
Screening 


Foliage Pen Syn App 
Rad 


Gridlock 


High Altitude Airship 


Joint Blue Force 
Situational Awareness 


Midnight Stand 
Night Vision Cave & 
Urban Assault 
Overwatch 


Tactical IFSAR Mapping 


Theater Support Vessel 


Tunnel Target Defeat 


Urban Recon 


Total 14 


FY04 ACTD 


Agile Rapid Global 
Combat Support 


Advanced Tactical 
Targeting Technology 


Coalition Reception 
Staging & Onward 
Movement 
Environment 

Future Tactical Truck 
System 

Joint Precision Airdrop 


System 


Control 


MAGNUM 


Joint ISR Interoperability 
Coalition 


Man Portable Threat 
Warning System 


Protected Landing and 
Take-off 


Global Reach 


Theater Effects Based 
Operations 


Total 13 


FY05 ACTD 


Actionable Situational 
Awareness Pull 


Operations System 


Chemical Unmanned 
Ground Reconnaissance 


Epidemic Outbreak 
Surveillance 
Engagement 

Joint Enhanced 
Explosion Resistant 
Coating Exploitation 


Joint Force Projection 


Theater 


Rapid Airborne 
Reporting & Exploitation 


Sea Eagle 
Sea TALON 


Demonstrator 
TACSAT-2 Roadrunner 
Gunship standoff 
Precision Munition 


Weapons Data Link 
Network 


Total 15 


FY06 ACTD 
Counterintelligence 
Architecture 
Modernization Progam- 


Intell Ops 


Management Planning 
Tool 


Awareness 


Event Management 
Framework 


Architecture 


Focused Lethality 
Munition 


Joint Enable Theater 
Access-Sea Ports of 
Debarkation 
Systems 

Large Data 

Counter Obscured 


Targets 


Node Management and 
Deployable Depot 


Capability 


Small Unmanned Aerial 
System 


Total 13 


FY07 ACTD 


Airborne Weapons 
Surveillance System 


Coalition Mobility 
System 


Global Observer 
IP Routers in Space 


Joint Multi-Mission 
Electro-Optical System 


Joint Surface Warfare 


Mapping the Human 
Terrain 


Maritime Auto Super 
Track Enhance Reporting 


Smart Threads Integrated 
Radiation Sensors 


Tactical Service 
Providers 


Total 10 


Table 2. ACTD Initiated by Fiscal Year (From: DoD Release 95-07) 
32 


D. CHAPTER SUMMARY 


This chapter began with a review of the ACTD process initiated in 1995. The 
background information provided the history on how the process got started. With that 
information we were able to provide detailed procedures on how an ACTD gets selected, 


funded, and eventually transitioned into the acquisition process. 


The most significant items associated with the establishment and methods of a 


given ACTD program include: 


1. ACTDs were initiated to reduce the acquisition cycle time and speed the 
delivery of advanced capabilities to the war fighter. 

2: The process utilized to establish and execute ACTDs has continually 
evolved since 1995 to increase the potential for success. 

Ds Technology maturity above readiness level 5, as defined in (Table 1), is 
critical to the expectation and ACTD will successfully transition into 
acquisition. 

4, Three categories of ACTDs exist: software systems, weapon and sensor 


systems, and system-of-systems. 


3: Exit paths available at the completion of an ACTD consist of: termination, 
return to technology base for further development, residual utilization, 
initiate acquisition at MS B with major improvements, initiate acquisition 
at MS C with minor improvements, initiate acquisition at FOC as COTS 
or NDI. The last of these is the desired goal of the ACTD process (South, 
2003). 
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ACTD TRANSITION PROCESS VS KNOWN PROBLEMS 


A. BACKGROUND 


Before we can understand the importance of planning for technology transition, 


we must first understand what technology transition means. Technology transition is the 


process by which technology deemed to be of significant use to the operational military 


community is transitioned from the science and technology environment to a military 


operational field unit for evaluation and then incorporated into an existing acquisition 


program or identified as the subject matter for a new acquisition program (Dobbins, 


2004). 


There are multiple challenges that are involved in the transitioning an ACTD into 


the normal acquisition process. Some of the issues that continue to surface throughout all 


the programs are: 


1. 


Contracting strategy this deals with getting the best deal from the 
contractor without losing momentum during the transition. 


Interoperability-is the ACTD compatible with other systems? 
Supportability-can it be supported in a cost effective manor? 


Test and Evaluation-getting these people involved as early as possible and 
keeping them involved throughout the transition is very important to the 
success of the ACTD 


Affordability-assessing life cycle affordability and application of a Cost as 
an Independent Variable (CAIV) strategy to continuously look for ways to 
reduce cost. 


Funding-finding the right strategy to get the resources for normal 
acquisition. 


Requirements-establishing a mission need and goals at the start and ending 
with system performance that captures the technology maturity and the 
knowledge gained by the war fighter in a realistic exercise. 


Acquisition Program Documentation-defining and planning for the 
documentation required prior to the acquisition decision (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004). 


aD 


Being able to address each one of these challenges will increase the probability of 
ACTDs transitioning into the acquisition model. Even though addressing all of these 
challenges does not guarantee a successful transition. The options for a failed program 
are to use it as an interim capability as is, continue development, or terminate the 
program. Having a solid plan in place way before the transition phase will help in the 
success (Office of the Deputy Under Secretary of Defense for Advanced Systems & 
Concepts, August 2004). 


Although the formal acquisition process is the primary means for procuring a new 
military capability the ACTD process is a pre-acquisition action that allows the war 
fighter to assess its military utility before going into full production. The transition will 
depend on the Military Unit Assessment (MUA) that an operational user will conduct. 
They will concentrate on the if the program is a valid requirement, that the maturity is 
sufficient for the purpose, and the ease of integration into a field usable product (Office 
of the Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 
2004). The ACTD Process is a pre-acquisition activity that is recognized by the 
acquisition system. Since FY95, there have been 167 ACTD programs. These programs 
are presented in Table 2. To date, 112 have completed the demonstration phase (due to 
the nature of ACTDs, those initiated FY06 or later are typically still underway). Of those 
completed, 16 have been terminated with another 3 placed on hold. There are 74 
programs that are reported as transitioned into the acquisition process while 19 are still in 
the transition phase. Out of all the years (excluding FY06-07), 51.4% have been coded as 
"transition on record" with an additional 13.2% in the transition phase. This is a 
combined total of 64.6% of the programs that are eligible (established for 2 or more 
years) to enter into the transition phase. Although these numbers do not seem impressive 
the process is making progress. A previous thesis project was done by Matthew South in 
2003. He reported that a total of 33% of the programs had entered into the transition 
phase at the end of FY02. This means that the success rate has doubled in the last 5 
years. For a fairly new program it seems to have the potential to continue on this path 


and reach over 80% success rate. 
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Table 3. ACTD Execution History (From: Carson Excel Spreadsheet, 2007) 


B. ACTD CLASSES 


ACTDs are categorized by three classes or categories. They are listed as 
Software/workstation/commo, Weapons, sensors, or C4ISR systems, and System of 
systems. These are generic classes of ACTDs that present significantly different 


transition challenges. 
1. Class I ACTD 


Software/workstation/commo (Class I ACTDs) is typically information systems 
with special purpose software operation on commercial workstations. They frequently 
are required in small quantities and that requirement can be satisfied without further 
development or production using the residual ACTD systems or a few additional systems. 
Class I is typically the easiest class to manage from a transition perspective (Office of the 


Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 2004b). 
2. Class I! ACTD 


Weapons, sensors, or C4ISR systems (Class II) are weapon or sensor systems 


similar in concept to systems that are acquired through the formal acquisition process. In 
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many cases a Class II ACTD will be planned to transition LRIP (post MS C) following 
the ACTD, but there may be cases where it is appropriate to plan for additional 
development following the ACTD (Office of the Deputy Under Secretary of Defense for 
Advanced Systems & Concepts, August 2004b). 


3. Class I ACTD 


System of systems (Class III) are an individual element within the overall 
architecture of a Class IIT ACTD maybe a fielded system, a system already in acquisition, 
or a system emerging from the technology base. The overall ACTD may involve 
multiple Program Executive Officers, and perhaps multiple Military Departments. The 
challenge may therefore be to integrate and coordinate the individual transitions to 
achieve the capability represented in the ACTD. Although existing ACTDs fit into each 
of the three classes described above, the only ones which have progressed to the point 
that a significant amount of transition planning effort has been performed are in Class II 
(Office of the Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004b). 


After completion, ACTDs have two possible exit paths - acquisition or non- 
acquisition. For the non-acquisition path the ACTD can be either terminated due to a 
lack of military utility, shelved for further development, or fielded to establish a residual 
operational capacity (South, 2003). Formal acquisition is based on the level of 
technology maturity demonstrated and MUA success. Based on these elements the 
acquisition can begin during System Development and Demonstration (SD&D), 
Production and Deployment (P&D), or additional elements can be procured for 


Operations and Support (O&S) (South, 2003). 
c. TRANSITION STRATEGY 


Technology transition into acquisition requires planning beyond that required for 


initial technology development. Acquisition programs involve a significant level of 
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oversight, planning and milestone reviews (Dobbins, 2004). Although this statement 
refers to the normal acquisition process, an ACTD program should not be treated any 


differently if the program is expected to transition. 


It is important that the transition into acquisition occur smoothly and without 
undue loss of momentum. It is critical that the objective be identified at the approval of 
the ACTD, and the transition strategy occurs during the early stages of the planning 
process which should be identified in the management plan (Office of the Deputy Under 
Secretary of Defense for Advanced Systems & Concepts, August 2004). As in most 
military transactions a poor plan will result in preventable issues to surface. These types 
of problems can usually be prevented when a solid, detailed plan is set in place. Starting 
the transition strategy late in the game will most likely cause the loss of momentum and 


possibly prevent the program from entering the normal acquisition process. 


1. Transition Plan 


Although there is no policy or other requirement specifying the contents of an 
ACTD transition plan, it should address elements specific to the technology being 
transitioned and how the technology will merge into the acquisition process of an existing 
program. The transition plan should be an element of the overall ACTD management 
plan and should reflect the transition strategy. It should address the transition issues and 
elements relevant to the specific technology being transitioned, including planning for 
operational user evaluations (Dobbins, 2004). ‘Transition planning is fairly straight 
forward but not at all easy. At the beginning of the ACTD, estimate whether the program 
will enter into the formal acquisition process. If entry is necessary the point at which the 
program will enter needs to be identified (MS B, MS C). After that a whole set of 
strategies need to be defined including contracting, supportability, interoperability, 
affordability, and requirements that will support the intended point of entry (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 2004). 
Another important piece of the puzzle is the implementation timelines for each of the 


above strategies. 
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a. Transition Integrated Product Team (TIPT) 


Part of an effective transition plan and management process is the 
formation-often by the ACTD demonstration manager-and activation of the transition 
integrated product team. The TIPT provides the most natural means for bringing the key 
stakeholders together to review strategies. It also serves as a bridge between the initial 
ACTD management planning activity and the transition decisions, assists in identifying 
and resolving transition issues. The receiving acquisition program office and the 


contractors should be included in the TIPT (Dobbins, 2004). 


b. Overarching Integrated Product Team (OIPT) 


As the time for completion of the ACTD approaches, an overarching 
integrated product team (OIPT) should be formed as a successor to the TIPT. The OIPT 
completes the remainder of the transition reviews (cost, schedule, and performance) in 
preparation for transition to acquisition. The OIPT ensures that all of the necessary 
elements and documentation are in place for the ACTD to transition into the acquisition 
program at the appropriate point in the acquisition life cycle. The OIPT will also prepare 


for a formal program review by the defense acquisition executive. 


It is also advisable (not required) to conduct a major review with the Lead 
Service organization that will be accepting both the interim capability assets from the 
ACTD and the objective system. This review should occur at least six months prior to 
the end of the ACTD and should address the status of preparations for operational support 
(Office of the Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004). Figure 4 shows how the transition strategy should flow throughout the 


process. 
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Figure 4. Transition Preparations (From: Carson PPT, 2006) 


Zs Contracting Strategy 


When a technology transitions into acquisition, there will be some form of 
contracting activity involved. The technology may be inserted into an existing contract 
whose terms and conditions will apply to the new technology. The program may enter 
acquisition as a major upgrade to and existing system and may require a separate 
contracting effort or it could enter acquisition as a major upgrade to an existing system 
that requires a separate contract. Another option is to have the project enter under a new 


contract effort (Dobbins, 2004). 


The initial contracting strategy should be based on the issues involved with a 
particular ACTD. It should consider the effort to be performed during the ACTD, as well 
the post-ACTD objective. A large part of the contracting strategy is deciding where the 
post-ACTD will enter into the acquisition process. For example, if the post-ACTD 
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objective is to enter directly into Low Rate Initial Production (LRIP), the contracting 
strategy should tailor the plan to enter production with the current design but allow for 
further development after the completion (Office of the Deputy Under Secretary of 
Defense for Advanced Systems & Concepts, August 2004). This portion should include 
how DoD will procure additional units of the ACTD if that is the decision at the 
conclusion of the phase. There are many decisions that need to be made within the 
contracting strategy. It has become obvious over the years that the early these decisions 


are made the chances of major problems later are reduced. 


The transition can be easier if the OIPT works with the acquisition community to 
ensure that a proper contracting strategy is in place. The contracting strategy must also 
make sense for the particular technology along with the existing acquisition process. 
There are many different contracting strategies, having a liaison with the acquisition 
manager for the program into which the technology will transition will significantly ease 
the transition process (Dobbins, 2004). In the end the contracting strategy alternatives, 
subsequent to the ACTD contract, must be specified in the solicitation. The possibility of 
continuing with the ACTD contractor into production should be clearly communicated to 
potential offers. DoD should be forthcoming within certain parameters of uncertainties 


that exist. 


3. Transition Funding 


RDT&E funding for ACTDs can currently be planned, programmed, and 


budgeted through two sources: 


1. Military Departments/Agencies 
2: OSD can supplement the service/agency to cover cost in three areas: 
a. added costs incurred when the technology programs are reoriented 
to support the ACTD. 
b. costs due to any requirement to provide additional quantities of 
hardware. 
C: cost for technical support for two year of field operations following 
the ACTD. 
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However, funding to support the follow-on activity (development, LRIP, full rate 


production, or purchase of additional quantities of commercial items) is not typically 


funded by OSD or the Service/Agency until the ACTD demonstrates military utility. 


This lack of prior funding creates a significant challenge that must be addressed as a part 


of the transition effort (Office of the Deputy Under Secretary of Defense for Advanced 


Systems & Concepts, August 2004). There are three follow-on funding strategies that are 


currently recognized: 


1. 


High Military Utility-No Resources programmed. When an ACTD has 
significant military capability but no resources have been provided to 
support the program. The lead service and present to the Defense 
Resource Board (DRB) a funding request. This will interrupt on-going 
programs funding by taking money away from another program in order to 
fund the ACTD. This type of funding strategy should only be used in an 
"urgency of need" situation that requires rapid acquisition (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004). 


Military Utility Established-No Resources Programmed. Lead service 
waits until the end of the ACTD to request funds. Since the POM is a 2 
year process the money is not available until 2 years after the completion 
of the ACTD. This means that the continuity from an ACTD to an 
acquisition program may be broken, and momentum lost (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, 
August 2004). 


Assume Success For Some ACTDs-Program Resources In Anticipation of 
Follow-On Acquisition. One way to avoid a break in the continuity 
between an ACTD and the follow-on acquisition program is to establish a 
budget line with funding, dedicated solely to the acquisition of the ACTD. 
This is a high risk move because if the ACTD does not prove military 
utility and is cancelled then the money set aside can not be used on 
anything else and would be lost. This would be normally done when 
military utility is expected to be high, and where there are early 
indications that the expectations will be met. If for some reason the 
program becomes joint, the lead service can transfer the resources to the 
Joint Lead Service for execution (Office of the Deputy Under Secretary of 
Defense for Advanced Systems & Concepts, August 2004). This strategy 
will prevent a break in continuity altogether. 
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D. ASSOCIATED RISK 


The overarching mantra of the ACTD process is increased rapidity in the delivery 
of relevant technology to the war fighter. In order to accomplish this, the procedures and 
regulations that have grown around formal acquisition programs have to be re-examined 
to determine bottlenecks within the process. One of the prevalent arguments against the 
formal acquisition process is the formality of the acquisition process (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, 2007). The 
belief exists that this degree of formality precipitates an unnecessary degree of oversight 
that bogs the process down. This fact, however, may not be as detrimental as it appears 


on the surface. 


In 1998, the Congressional Budget Office (CBO) issued a report on ACTD 
programs (Congressional Budget Office, 1998). Within the report, the CBO dug below 
the surface issue of apparent superfluous oversight and identified one of the subcutaneous 
(and unforeseen) effects of stripping back control —it introduces further risks into the 
ACTD process. ACTD programs, by their nature, are conceived with inherent technical 


risk but lack of proper management introduces a new subset of risk. 


1. Questionable Project Selections 


One of the harshest criticisms leveled against ACTD programs in the report 
(Congressional Budget Office, 1998) was in regards to the selection process. The 
overarching concern with this risk was that an improper selection of an ACTD that was 
not as technologically viable as another alternative would become more difficult to 
manage. Because ACTDs fall outside the realm of traditional acquisition processes, the 
ability to mitigate risk through more structured reviews is degraded within an ACTD 
program. This inability to grasp and reel in technological difficulties has the potential to 


translate into significant cost overruns. 
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2. Ambiguous Criteria in the Selection Process 


The risk drawn out by the CBO (Congressional Budget Office, 1998) is not one 
that berates the selection process of ACTDs per se; instead, it addresses how ACTD 
program offices interpret two constraints bounding the selection process. The first 
boundary drawn out is the degree of technical maturity. The main critique of the 
technological maturity is a precise definition of what it means for a program to be 
sufficiently mature. GAO notes that their interpretation of maturity within DoD is a 
system that has demonstrated successes at the subsystem or component level (United 
States General Accounting Office, 2002). They do not denote what constitutes a 
successful performance. The lack of fidelity within the definition has led to the adoption 
of many definitions of technology maturity within ACTD processes. DoD has tried to 
mitigate this situation by stating that ACTD programs be assessed on their maturity with 
the same TRLs used by traditional acquisition programs (United States General 
Accounting Office, 2002). Another weakness noted is the seeming lack of understanding 
on when to apply the notion of maturity to an ACTD program (Congressional Budget 
Office, 1998). The issue stems from a propensity for program offices to look toward the 
ultimate product that an ACTD program provides to the user; however, this lack of focus 
on present program maturity creates a disparity between determining if a current 
technology is sufficiently mature and designing a path to an end product. In other words, 
a lack of understanding on the degree of maturity on a current program precludes a clear 


roadmap because program offices focus a great deal of effort on the end product. 


The second parameter noted by the CBO (Congressional Budget Office, 1998) is 
a disparity of ACTD development and assurance that they are addressing an urgent 
requirement. The issue brought to light is that, though the JROC provides initial 
screening of candidate programs for the ACTD process they are not active participants in 
the transition process (United States General Accounting Office, 2002). This lack of 
uncertainty on the joint utility of a program may be one of the factors prohibiting ACTDs 
from transitioning into the formal acquisition phase. Because the JROC does not have a 


role in the transition process of ACTDs, these programs often lose upper-level support. 
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Therefore, the transition process falls back to the services which often do not support the 
ACTD program because there are other priority programs or the ACTD extends beyond 
their primary mission. 

As identified by the Deputy Under Secretary of Defense for Advanced systems 
and Concepts (DUSD(AS&C)), which is supposed to be used by ACTD managers and 


users, the following list of criteria are used to determine approval/implementation of a 


purposed ACTD program: 


Criteria 
1. The ACTD met a priority military 


Indicators of Success 


a. The proposed solution incorporated intense user involvement to evaluate the ability to 


need. (C-1) 


. The ACTD was adequately mature, 
(C-2) 


The ACTD demonstrations / 
exercises were adequate to provide 
a military unity assessment. (C3) 


. The ACTD developer 


demonstrated the essential ACTD 
entema. (C-4) 


. The ACTD lead service executed 
the program appropriately, (C-5) 


. ACTD sponsorship was executed 
appropriately. (C-6) 


. The ACTD execunon window 
completed withim two to four 
years. (C-?} 

. The risks associated with the 
ACTD were appropnately 
identified. (C-8) 


meet military needs. 


. ACT Ds users had realastic and extensive mulitary exercise opportunities to evaluate 


unlity and gain experience with capabilities, 


. Users refined their operational requirements, developed CONOPS, and developed a 


sound understanding of the mulitary utilaty. 


ACTD was a mature of nearly mate technology based on Technology Readiness 
Level (TRL) scale (minimum level 5 or above). 


b. ACTD marurity reduced the time and msks associated with the demonstration. 
. ACTD activities focused on mtegration and demonstration activities not technology 


development. 

Adequate quantities of ACTD were procured to provide a valid assessment of its 
capabilities. 

ACTD demonstration was sized and stmictured to provide a clear evaluation of nuilitary 
capability. 


. ACTD integrated / executed both developmental and operational TRE swiftly and 


economically to ensure that requirements were met and the system was operationally 
satisfactory and usefull. 

The user defined the MOEs and MOPs that allowed effectiveness and suitability to be 
characterized. 


. User plamsed the operanonall exercises, typically including red and blue forces. 
. The potential or projected effectiveness was sufficient to warrant consideration as an 


ACTD. 
The available capability addressed a need for which there was no suttable solution. 


. The abality of the technology to be interoperable with other systems on the battlefield 


was Werihed 


. The fielded system would maimtaim a high state of readiness and safety, using traaned 


operators add maistainers, and do so economucally and with the smallest possible 
logistical footprint. 


. The lead service/agency ensured the necessary planning for transition to formal 


acquisition was accomplished 


. The lead service/agency ensured transition of the residual assets to the user 


organization and for all aspects of thei support. 


. The JROC recommendation for lead service/user sponsor was accepted by 


DUSD(AS&C). 


b. The user sponsor was a Unified Commander (general rule, not as a requirement). 
. Affordability goals were set for acquisition amd life-cycle costs that permitted CAIV 


trade-offs and later design-to-cost (DTC) tradeoffs, 


. The ACTD completed all activites within the expected time (2 - 4 yrs). 
. No significant schedule or configuration changes were recyuired to meet objectives. 


. Risks were identified and accepted by the primary stakeholdets of the ACTD. 
. Programmatic nsks (¢.g. cost and schedule) and the operational risks related to the 


acceptability of the operational concepts necessary to realize the full benefat of the 
proposed capability were minimized. 





. System complexity (low to high) was related to nsk level (low to high), respectively. 
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9. ACTD funding was sufficient to . Abudget was developed and submitted as a past of the proposed ACTD. 

meet program requirements. (C-9) _ The proper strategy was chosen for obtaining the resources necessary for acquiring the 

technology 
. All costs associated with the design and developnyent of the prototype system was 

identified (this mcludes all additional units required in the ACTD, all exercises that 
must be paid for by the progect, and test support casts including any modeling 
simulation and analysis needed to support the utility assessment) 
The ACTD budget ineluded transition costs related to the planning and preparations for 
acquisition, as well as the cost to provide technical support for the first two years of 


fielding the resiciuals 


10. The ACTD executed its transition | a, The ACTD did not lose momentum in transitioning to the acquisition process 
plans to inatiate acquisition. (C-10) (assuming the user made a positive determination of mulitary utility). 

A clear acquisition goal was set for the post ACTD phases 

. Requirements were evolved from mussion need and performance goals to formal 
operational requirements documents; unteroperability documents; system performance 
specifications; and total ownershap costs (manning, training and sustainability related 
to applying the technology) estimates. 

. A cantracting strategw was established that motivated the contractor to provide a best- 
value solution (an terms of overall life-cycle cost-effectrveness) and permutted 
transition ante procurement without the loss of momentum. 





Table 4. | ACTD Criteria and Indicators of Success (From: ACTD Guidelines, August 
2004) 


3. Too Little Oversight 


One of the risks identified by the CBO (Congressional Budget Office, 1998) was 
a concern that the ACTD may be circumventing the acquisition process and not 
supplementing it. Restrictions are in place that restricts the number of prototypes bought 
only to those necessary for operational testing (Mol, 1998); however, ambiguity 
surrounds what is necessary for operational testing and for how long. The concern is that 
an ill-defined operational testing environment could involve extensive field trials with an 


undo number of test vehicles. 


In addition to a proper of mix of test vehicles, Congressional concerns also 
surround the degree of stringency applied to ACTD contracting practices (Congressional 
Budget Office, 1998). Again, due to their operations outside the traditional acquisition 
process, ACTDs are open for interpretation in regards to contracting procedures. There 
appears to be no fast procedures (only guidelines) for the application of contracting 


criteria when dealing with ACTDs. 
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4. Future Demands for Defense Spending 


The final problem noted by the CBO (Congressional Budget Office, 1998) is that 
due to their less- formalized nature, ACTDs often do not have the necessary cost 
projections available if an ACTD transitions into the formal acquisition process. This is a 
direct outgrowth of the uncertain nature of ACTDs in their initial development. An 
ACTD lives in a constant state of termination or reversion back to a lab environment, 
therefore, it is not imperative that long-range financial forecasts are formalized in the 


early stages of the program. 


If an ACTD makes it past the “halfway point” in the ACTD process, the guideline 
is for the transition integrated product team (IPT) to begin developing costs associated 
with acquiring the program. Long-term plans, however, regarding ACTDs are inherently 
uncertain and little certainty can be assigned to cost analysis data generated in the long- 
term for the program. This uncertainty in costing sets up a scenario where ACTDs are 
funded with a chunk mentality (Congressional Budget Office, 1998) which may or may 


not prove sufficient to properly develop the program. 


This lack of adherence to formalized fiscal planning processes may be one of the 
factors jeopardizing the unwillingness to fund an ACTD’s transition (United States 
General Accounting Office, 2002). The main issue is that the lack of authoritative cost 
projects does not allow for inclusion in the formalized budgeting process. What is often 
the case is that services must now take funds from their own coffers to support the 
transition of the program. As was previously mentioned, the lack of support generated 
for these programs by the services can prevent programs from receiving funding for 


transition. 
E. CHAPTER SUMMARY 


This chapter has addressed the transition process of the ACTD programs, 
including the challenges that each program faces each time they enter into the process. 
The execution history has identified all the programs to date and where they stand in the 


process. This helps identify where the successes and weaknesses are located in the 
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process. We discussed the different ACTD classes along with which class is the most 
likely to proceed through the programs with the least amount of challenges. Identifying 
the importance of the transition strategy is a key element. Without the early planning of 
this strategy the likelihood of a program entering into the normal acquisition process is 
very limited. How the TIPT and OIPT play key roles through out the process but more 
importantly in the contracting strategy phase. Funding is probably one of the most 
important phases of the transition. Without the proper funding in place all the planning is 
irrelevant. Finally, we reviewed known problems with the ACTD process from project 


selection, ambiguous criteria, down to the funding issues involved with the process. 
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IV. ACTD ANALYSIS 


This chapter will describe the approach employed for an overall analysis of those 


ACTD programs for which data was provided. 


The initial section will demonstrate the methods to both summarize and codify the 
data contained with the ACTD management plans. The data gleaned from the program 
management plans was not standardized and had to be encapsulated and encoded for 
effective analysis. To understand this, the approach to analyze the encoded data will be 


explained. 


Once an understanding of the data and encoding techniques are established, a 


brief overview of the statistical model employed will be discussed. 


Finally, given the outputs from the statistical model, possible causal relationships 


will be examined. 
A. DATA SUMMARY 


In our analysis we limited our investigation to one quadrant of the two-by-two 
diagram (upper right) below (Table 5). The quadrant we investigated involved 
information provided within the management plan (vertical axis) and a hypothesized 
effect (horizontal axis) from the given data set. Others could look at this by using a 
different source but the same variable (lower right), using a different source different 


variable (lower left), or same source different variable (upper left). 


The initial challenge when presented with the provided ACTD data was the 
application of a systematic approach to summarize the data. The data we examined 
pertaining to the ACTD programs was the program management plans. Though the 
management plans contained information pertaining to a myriad of aspects of the ACTD 
programs, they were not uniform in their approach or depth. Therefore, the approach to 
data summarization was to identify all aspects of the ACTD management plans that were 
a) available for review and b) might possibly affect the final disposition of the program. 


Two important limitations to the study should therefore be mentioned here. First, the 
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data source we used is a limitation in the sense that we were limited to coding what data 
was available (or missing) across management plans. Second, we used our own 
hypotheses about which factors might possibly affect transition, and were guided by our 
advisors knowledge of prior work on technology transition. What emerged from this 


coding process was a set of coded variables that we believe is reasonable and robust. 


In order to ensure uniformity, each management plan was reviewed for those 
program facets that were identified within the document. Once a comprehensive list of 
readily available program attributes were delineated from the collection of management 
plan scrubs, a crosscheck identified those attributes which were common throughout the 


entirety of the plans. 


When the listing of common attributes was finally compiled, it was reviewed for 
possible significance. The information was subdivided into two categories. One 
category was information that was available for all programs but was reckoned 
inconsequential to program outcome. The second category was information that was 


available for all programs and was reckoned consequential to program success or failure. 


Yes 


No 


UD]d 18M U1 ajqnjiway vIVG 


No Yes 
Hypothesized Possible Affect 


Table 5. 2 x 2 Diagram (From: Wideman/Phelps, 2007) 
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From the list of substantial data shared by all programs, eleven questions were 
constructed to identify those areas and ancillary aspects of those areas for analysis. 
Before the questions could be put to the management plans, however, a system of coding 


the data needed to be established. 


B. CODING 


The coding applied to the data (and serving as the basis for analysis) fell into 


three distinct categories. 


The first category was binary. If the question lent itself to a simple yes or no 


response, this approach to coding was used. 


The second category was a scaling system. This approach provided an approach 
to coding those questions that were more subjective in nature. Not surprisingly this 
approach constituted a large portion of the questions. Underlying the scaling process was 
an attempt to create a consistent and objective evaluation of variables. Our intuition is 
that program management provides a consistently optimistic forecast for program success 
no matter the underlying difficulties inherent to the program. Scaling the responses 
allowed the data presented in the management plan to strengthen program management’s 


hopeful outlook against a more objective approach. 


The final category was number entry. This approach was utilized in those 
instances where actual numbers were either provided or could be determined by some 
rudimentary analysis, and the two aforementioned methods were not appropriate for 


coding. 


We examined the following variables. 


1. Transition, or Not 


This was the dependent variable in our analysis of the thirty-eight programs. Of 


the programs studied 19 of them transitioned into the acquisition process while the other 
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19 did not. A binary method was employed to mark each ACTD as transitioning or not. 
A "1" was assigned to those programs that transitioned and a "0" to the non-transitioned 


ACTDs. 


We coded the following independent variables. 
2. Manager Assignment 


Per ACTD Guidelines: Management Plans (Office of the Deputy Under Secretary 
of Defense for Advanced Systems & Concepts, 2007), by the point that a management 
plan is produced for a given ACTD, the guidance states that managers of the project must 
be identified. The extent of management identified is not explicit but implies all 


anticipated management will be named. 


Seven key members of program management (IPT manager, oversight group, 
executive agent, user sponsor, technical manger, transition manager, and operations 
manager) were selected for identification. Once the positions were determined, the 
management plans were reviewed for name identification. If the plan provided a specific 
point of contact, the corresponding position was assigned a code of “1,” marking it as 
having met identification criteria. When the plan was reviewed and specific members 
were identified, the management members were summed to present a total number of 


those members actually assigned out of 7. 
3 Budget Matched to Schedule 


In order to determine the correspondence between schedule and forecasted 
funding data, the two provided schedules were compared for discrepancies. A 
rudimentary exam of the two schedules often noted minor discrepancies that may have 
negated this metric; however, further scrutiny identified finesse in budgeting and 
development scheduling that required investigation. Many of these programs were 
overlapping the provided funding by extending the program schedule past a noted 
funding timeline. A closer examination noted that many of these efforts were coming 


from further auxiliary funding sources or were play in the schedule for further 
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development. Though the charted timeline provided an initial conflict at first glance, the 


written narrative delineating the development effort often alleviated such disparities. 


Given this information, we coded plans with a binary code (1.e. 0/1) depending on 
whether there was a reasonable match between the development schedule (both written 


and charted) against the funding matrices provided in the management plan. 
4. Established Military Need 


The determination of an established and justified military need necessitated the 
use of a scaling system. When the management plans were compared across one another 
it became apparent that a binary coding system was wholly inadequate in measuring this 
program facet. The elaboration and citations contained within this portion of the 


management plan lent itself to scaleable grading system. 


At the lowest end of the scale was a low rating. The low rating reflected either a 
complete lack of inclusion of the military need justification or a generic justification that 
had a few specific connections to the particular ACTD program. Typically a need was 
posited based on a strategic or tactical deficiency, but little or no effort was applied to 


demonstrating how the ACTD contributed to fulfilling the claimed need. 


The middle rating was a medium rating. A medium rating demonstrated an effort 
to tie the ACTD program to a particular military need; however, it did not demonstrate 
conclusively how the particular program would help remedy the situation. Within this 
rating, a continual tendency of the management plans was to generalize the military need 


and hypothesize how the ACTD could be molded to meet this need. 


We coded plans "high" when the management plans identified specific military 
failings that facilitated the overarching military need and explicitly addressed how the 
particular ACTD in development would fulfill this need and showed a clear path on how 


to achieve this aim. 
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5s Technology Maturity 


One of the most difficult program parameters to gauge was the level of 
technology maturity of the ACTD. The difficulty arose primarily from the degree (or 
lack thereof) of data provided within the management plans regarding technology 
maturity. Traditionally, the maturity of technology is spelled out in a regulated system 
known as Technology Readiness Levels (TRL) (Advanced Systems & Concepts, 2006). 
However, the management plans provided were inconsistent in their mention of TRL’s 
pertaining to their programs or did not provide enough technical data to ascertain a rough 
estimate on where the program may fall on the scale. Another method of scaling was 


needed. 


The foundation of most of the ACTD’s relied heavily on the success of software 
development. The thread of software success ran through all of the management plans 
and provided a basis for some scaling of technology maturity. In lieu of TRL’s, an 
alternative system for gauging technology maturity, known as ImpACT, was utilized 
(Advanced Systems & Concepts, 2006). ImpACT deals specifically with software 
development and the degree of impact of commercial-off-the-shelf (COTS) software on 
the success of the development effort. ImpACT allowed for an effective scaling 


methodology to be developed based upon the description of development. 


The highest rating was an all COTS/pre-existing rating. Given the lack of detail 
within the preponderance of management plans, this rating reflected an assignment to 
software (technology) that they had been developed and tested and demonstrated to be a 
viable option. It does not reflect the anticipated degree of success of militarizing the 
COTS but simply shows that the technology is one in existence with some degree of 


practicality. 


The middle rating was partial COTS rating. This rating demonstrated that at least 
part of the software (technology) development effort was more mature than new 
development due to the inclusion of COTS. It also took into account that because the 
effort was partially composed of new development (in addition to COTS) the technology 


maturity was lessened. 
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The lowest rating was the new rating. Reflected within this rating is the 
knowledge that the software (technology) has no aspect that has either been fully 


developed or tested. 
6. Transition Strategy 


The transition strategy was also coded using a scaling metric. Like the 
aforementioned military need criteria, transition strategy’s scale had to be constructed, 
not against an available benchmark, but against the verbiage provided within the 


management plans themselves. 


A low rating reflected either a complete omission of a transition strategy or an 
acknowledgment that it was the hope of the program office that the program would 


transition at a later date. 


The medium rating was given to those ACTD efforts which had a transition plan 
in development. In other words, the program management plan had either a detailed plan 


to develop a transition plan or had a transition plan lacking significant detail. 


The highest rating was given to transition plans that had exacting details (e.g., 
involved parties, sequential processing, timelines, etc.). This rating reflected a transition 


plan that marked a clear path to incorporate the ACTD beyond the development phase. 
7: Timeline Requirement 


Per ACTD Guidelines: Formulation, Selection and Initiation (Office of the 
Deputy Under Secretary of Defense for Advanced Systems & Concepts, August 2004), 
ACTD programs are intended to have a development and demonstration lifetime of 
between two and four years. This timeline underscores the primary presupposition for 
ACTD programs to be rapid development efforts to get critical technologies to 


combatants. 


For the supplied management plans, the development and demonstration 
schedules were reviewed for number of years. The years were counted and actual 


numbers were entered. The numbers entered do not reflect ancillary efforts that may 


a 


have preceded the ACTD. For consistency, if such efforts were noted on the schedule, 
we discounted them and did not include them in the overall count. Given our sample size 
(over 30) and random selection of our sample, it is reasonable to assume that we have 
randomized for ancillary efforts, and that these should not bias our statistical analysis in 


any particular way. 


8. Plan Depth 


Plan depth was coded with actual page counts of the management plans. The 
assumption here is that increased page counts are indicators of a more developed 


management plans. 


9, Number of Parties Involved 


For the number of parties involved, actual counts of the parties were used. There 


were two considerations made in ascertaining party involvement numbers. 


The first consideration is that one mention reflected one count. Many of the 
involved parties were mentioned under numerous development phases, as well as, 
numerous integrated product team (IPT) membership. Second we considered how ACTD 
management is intended for implementation. The program is subdivided into sub-efforts 
(e.g., IPT). Each sub-effort is managed by an overarching agency appointed by ACTD 
program management. The purpose of the underlying management is to align and rectify 


concerns within their functional area. 


It might be noted that the number of parties involved is an indicator of the 
coordination complexity involved in a particular ACTD program. More complex 
coordination arrangements may be reasoned to be more prone to breakdown, and 


therefore to failure for a program to transition. 


10. Capital Investment Requirement 


This metric is summation of budgeted dollars laid out in the management plan. 
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11. Technology Complexity and Risk Assessment 


These last two metrics were coded separately. However, they are intrinsically 
related and will be discussed together. Technology complexity was not a metric included 
in the original management plans. It was a subjective measurement we included in our 
consideration of the overall complexity of the program. Each was coded as high, 


medium, or low. 
First we coded technology complexity. 


1. We looked at the scale of proven technologies versus unproven 
technologies within the program. 


2 We then verified whether or not a clear plan of technical integration was 
present. A clear plan was considered one that addressed an obvious path 
for integrating the technologies as a whole to accomplish the function of 
the program. 


3. Finally, if the management plan mentioned utilization of COTS but also 
noted a lack of forecasted integration between COTS programs, we coded 
it as complex. We assigned a subjective rating of high, medium, or low 
based upon these three factors taken as a whole. 


Second, we coded plan risk. 


1. Management plans that identified risk in a program usually identified it as 
low-to-medium risk. Not all management plans included their own risk 
assessment and, given the criticality of a risk analysis in a development 
program, we included a subjective risk analysis. We coded these and 
included them in our regression analysis. 


2: We coded plans based on specific mentions of risk analysis and risk 
mitigation efforts or procedures that were being put in place. We 
considered information contained in the management plans on risk 
mitigation efforts in place or being put in place as part of the management 
plan for the particular ACTD in question. 


a We then coded the management plans based on our own assessment of 
their risk level. In part, the risk measure we used reflects our assessment 
of the nine aforementioned metrics. This metric was utilized to provide an 
independent and (relatively) objective assessment outside of program 
management authorship of the overall program risk, given the data 
provided in the management plan. 


59 


C. METHODOLOGY 


After we coded the variables we analyzed them using multivariate regression so 


as to examine their relationship with ACTD transition. 
1. Multivariate Regression Analysis-Explanation 


Multivariate regression can establish that a set of independent variables explains a 
proportion of the variance in a dependent variable at a significant level (through a 
significance test of R’), and can establish the relative predictive importance of the 
independent variables (by comparing beta weights). Power terms can be added as 
independent variables to explore curvilinear effects. Cross-product terms can be added as 
independent variables to explore interaction effects. One can test the significance of 
difference of two R”'s to determine if adding an independent variable to the model helps 
significantly. Using hierarchical regression, one can see how most variance in the 
dependent can be explained by one or a set of new independent variables, over and above 
that explained by an earlier set. Of course, the estimates (b coefficients and constant) can 
be used to construct a prediction equation and generate predicted scores on a variable for 


further analysis (Garson, 2007). 


Dummy variables are a way of adding the values of a nominal or ordinal variable 
to a regression equation. The standard approach to modeling categorical variables is to 
include the categorical variables in the regression equation by converting each level of 
each categorical variable into a variable of its own, usually coded 0 or 1. For instance, the 
categorical variable "region" may be converted into dummy variables such as "East," 
"West," "North," or "South." Typically "1" means the attribute of interest is present (ex., 
South = 1 means the case is from the region South). Of course, once the conversion is 
made, if we know a case's value on all the levels of a categorical variable except one, that 
last one is determined. Why not run separate regressions? It is true that one approach to a 
categorical variable in regression would be to run separate regressions for each category. 
While this is feasible for a single variable such as gender, running a male and a female 
regression, it is not the best approach for two reasons. In practical terms, if there are 


multiple categorical variables each with multiple categories (levels), the number of 
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needed regressions may become unwieldy. In statistical terms, we will lose power since 
each regression will have a smaller sample size than if there were one overall regression. 
That is, we will be more likely to make Type II errors (false negatives, thinking there is 


no relationship when in reality there is) (Garson, 2007). 
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Table 6. | Raw Data Coding (From: Wideman/Phelps Analysis, 2007) 
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D. ANALYSIS 


1. Overall Model Results 


The full model (Table 7) indicates a coefficient of determination adjusted for 
degrees of freedom (Adjusted R2) as .40. This would suggest that 40% of the model is 
explained by the 11 independent variables. When the core model (Table 8) was run with 
the three statistically significant variables (from the initial full model), the adjusted R2 
climbed to .50. This indicates that the model with these three independent variables 
explains 50% of the variance. The output of the regression model noted several distinct 


areas with noticeable impact on the resolution of an ACTD program. 


SUMMARY OUTPUT 


Regression Statistics 


SS MS F Significance F 
Regression 5.55483 0.50498 3.32802 
Residual 3.94517 0.15174 
Total 25 


Coefficients StandardError tStat P-value Lower95% Upper 95% Lower 95.0% Upper 95.0% 
Intercept 0.99265 0.63874 1.55407 0.13226 -0.32031 2.30560 -0.32031 
Budget Matches Schedule 0.72701 0.19680 3.69424 §ONO0T03! 0.32249 1.13153 0.32249 
Technology Maturity -0.38282 0.17502 -2.18728 (003797 -0.74258 —_-0.02306 -0.74258 
Risk Assessment -0.29174 0.17528 -1.66446 [OM0803) -0.65202 0.06855 -0.65202 
Capital Investment 0.00041 0.00048 0.85332 0.40127 -0.00057 0.00138 -0.00057 
Transition Strategy 0.06007 0.11339 0.52980 0.60074 -0.17300 0.29315 -0.17300 
Military Need Established 0.07130 0.13854 0.51464 0.61115 -0.21347 0.35607 -0.21347 
Page Count (Plan Depth) -0.00289 0.00627 -0.46013 0.64925 -0.01578 0.01001 -0.01578 
2-4 Year Requirement 0.06853 0.15948 0.42974 0.67092 -0.25928 0.39635 -0.25928 
Technology Complexity 0.05759 0.14643 0.39328 0.69732 -0.24340 0.35858 -0.24340 
‘Total Managers Identified -0.01249 0.05208 -0.23991 0.81228 -0.11954 0.09455 -0.11954 
Parties Involved 0.01112 0.05353 0.20774 0.83705 -0.09892 0.12116 -0.09892 





Table 7. | Regression model for Significant Variables Only (From: Wideman/Phelps 
Analysis, 2007) 


Several authors have offered guidelines for the interpretation of a correlation 
coefficient (Statsoft, 2003). As Cohen himself has observed, however, all such criteria 
are in some ways arbitrary and should not be observed too strictly. This is because the 


interpretation of a correlation coefficient depends on the context and purposes. A 
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correlation of 0.9 may be very low if one is verifying a physical law using high-quality 
instruments, but may be regarded as very high in the social sciences where there may be a 


greater contribution from complicating factors (Statsoft, 2003). 


Ze Independent Variables that are Significant Predictors of ACTD 
Transition 


The output of the regression model noted three distinct variables that had an 


impact on the resolution of an ACTD program. 


We examined these in a second regression model — Figure 6. This model has an 
overall adjusted R2 of 0.50, which means that these 3 variables can be used to generate a 
simple model that is a strong predictor of ACTD transition results, i.e., here is a simple, 


parsimonious set of predictors. 


These areas are discussed below in an attempt to garner an understanding of the 


causal relationships that might explain the statistical results we found. 
a. Budget Matched to Schedule 


The first variable that was statistically significant was “budget data 
matched to schedule.” Those programs that progressed successfully were more likely to 
have funding available to match program scheduling. Conversely, those programs which 


failed to successfully progress were more likely to have the opposite relationship. 


Given the importance of a program’s schedule and the supporting funding 
means, proper alignment between the two is essential. The data presented in the 
management plan stage of ACTD programs demonstrated that this consideration was not 


always rectified satisfactorily. 


The correspondence between a properly aligned budget and schedule and 
the ultimate success or failure of an ACTD program is notable for several possible 
reasons. The failure to align two program facets so inherently related demonstrates an 
absence of consideration for one or the other when constructing the profile. This lack of 


communications could be facilitated through poorly aligned communications methods or 
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a lack of congruence when developing the management plan. Overall, it portrays a 
potential instance where these two facets are developed with either minimal consideration 


or concern for the other. 


Given the omnipresent status of an ACTD’s budget it is hard to justify that 
this is as prevalent as the data presents. What this relationship presents (in addition to 
lack of consideration) is a degree of living beyond current constraints. A program 
schedule contrary to funding profile projects a sense of a project lacking the proper 
understanding to align the disjointed facets. The technical schedule is held as a 
projection to achieve a viable program and an inability to mold it to funding constraints 


demonstrates a questionable understanding of the program as it currently stands. 


b. Technology Maturity 


Another area cited with high significance is the degree of technology 
maturity. The more mature an ACTD’s technology was (heavily cited as COTS in this 


instance) the higher was the probability of success for the program. 


The correlation between technology maturity and program success is 
important in the context of an ACTD’s existence. The ACTD program exists to rapidly 
apply technology to a real-world concern. In other words, it is about the proper 


application of technology. 


ACTDs either utilize existing technologies in a new manner, mix new and 
existing technologies to augment current capabilities, or develop new technologies. The 
issue is that simultaneous development is occurring and the more certainty (or proven 
ingredients) that can be included in the outset, the focus on the solution becomes more 
salient. While the utilization of technology addresses the technical aspect of a real-world 
military issue, there is also the tactical concern of how to employ said technology. The 
more proven a system’s underlying technology is allows for more rapid evolution to a 


war fighter's demands while concurrently not promising more than can be provided. 
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c. Risk Assessment 


The final area with a reasonably high significance is the risk assessment. 
Those programs coded as lower risk were more likely to succeed. In many ways, this 


corresponds to the abovementioned conclusion drawn from technology maturity. 


One additional consideration here (in addition to technology maturity) 1s 
the certainty that an overall risk assessment can be inferred from. When management 
plans consistently they demonstrate a propensity towards marking their own programs as 
having low technical risk opens the forum for more stringent observation. Those plans 
that have failed to provide a thorough analysis have opened themselves up for the 


possibility of termination or a serious degradation in their support. 


SUMMARY OUTPUT 


Regression Statistics 


Regression 3 5.16616 1.72205 13.50994 
Residual 34 4.33384 0.12747 
Total 37 9.5 


Coefficients Standard Error tStat P-value Lower95% Upper 95% Lower 95.0% Upper 95.0% 
Intercept 1.17339 0.55039 2.13191 0.04032 0.05486 2.29192 0.05486 
Budget Matches Schedule 0.77935 0.14193 5.49106 §§ONO0000) 0.49091 1.06779 0.49091 
Technology Maturity -0.36725 0.14892 -2.46604 |OIOTS86) -0.66990 -0.06460 -0.66990 
Risk Assessment -0.21433 0.13300 -1.61151 0.11631 -0.48461 0.05596 -0.48461 





Table 8. | Regression Model for Significant Variables Only (From: Wideman/Phelps 
Analysis, 2007) 


3. Correlation between Significant Variables — Could we Simplify the 
Model Further by Eliminating Anything Else? 


Once the significant variables were identified we looked at the correlation 


between the three. 


The three variables that we found to be correlated were technology maturity, 
Budget matches Schedule, and risk assessment. These three had a positive correlation. 


The relationship between three variables is such that as one variable's values tend to 
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increase, then the other variable's values also tend to increase. This is represented by a 
positive correlation coefficient (Statsoft, 2003). We also found that some of the variables 
had negative correlations. This means that the relationship between two variables is such 


that as one variable's values tend to increase, while the other variable's values tend to 


decrease. If the budget increases then the maturity level would tend to decrease (Statsoft, 


2003). 





Total Budget Military 
Managers Matches Need Technology Transition 2-4 Year Parties Technology Risk (Plan Capital 
Identified Schedule Established Maturity Strategy Requirement Involved Complexity Assessment Depth) Investment 
‘Total Managers Identified 1 
Budget Matches Schedule 0.55817 1 
Military Need Established 0.43561 0.43503 1 
Technology Maturity 0.32664 §NOM750) -0.03499 1 
Transition Strategy 0.34858 0.29011 0.66989 -0.05051 1 
2-4 Year Requirement 0.07191 -0.04942 -0.20404 0.35085 = -0.19280 1 
Parties Involved 0.54582 0.32021 0.44129 0.05774 0.39214 -0.26816 1 
Technology Complexity -0.08823 -0.31208 0.25035 -0.52671 0.17046 -0.13318 0.01922 1 
Risk Assessment -0.35662 NOM7730) -0.02527 NNOW7I22 «(0.07443 -0.24736 -0.10886 0.67607 1 
Page Count (Plan Depth) 0.59889 0.49521 0.63518 0.09658 0.65794 -0.12157 0.46631 -0.07085 -0.21105 1 
Capital Investment 0.26217 0.16029 0.02544 0.04517 0.18892 -0.20567 0.13642 -0.01376 -0.03373 0.09782 
Figure 5. Correlation Table from Data Set (From: Wideman/Phelps Analysis, 2007) 
4. Non-Significant Variables 


Non-significant results for variables are just as important as significant variables. 
There are two particular variables that were non-significant that are especially worth 
mentioning. Though the technology maturity was deemed to be significant to the 
determination of an ACTD’s success prospects, technology complexity did not. What 
may be underlying this phenomenon is the coding method chosen. While technology 
maturity lends itself to more objective coding, technology complexity is a subjective 
measure applied in consideration of all programmatic features. Through an attempt to 
summarize the program in a single metric, the metric mirrors the program. By not 


singularizing within the metric, the program’s success is inherently tied to the metric. 


In addition to the occurrence of technology complexity is an issue of parties 
involved. The negligence of this metric to play a critical role in a program’s ultimate 
resolution is born out of management foresight from the ACTD program office (and is 
hinted at in the coding process). Though a multiple players may be involved in a 


program’s development, their potential disputes may have been are filtered by the time 
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the management plan is in place, 1.e., the management plan may be considered an 
indicator of dispute resolution, without which it would have never been completed. This 


might explain the non-significance of this variable. 
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V. 


RECOMMENDATIONS AND CONCLUSIONS 


A. RECOMMENDATIONS 


After comparing 38 programs of which half transitioned into the normal 


acquisition process we have found that some variables within the management plans of 


ACTDs are powerful predictors of whether ACTD programs will transition or not. These 


results suggest recommendations to make the ACTD process more successful, as follows: 


1. 


Ensure that the over all technology maturity level is high. This means the 
appropriate interoperability of all the sub-systems must achieve a certain 
maturity level. Having COTS sub-systems is a good start but without 
bringing them all together to prove military utility will be less favorable to 
transition. Earlier in this paper we talked about Technology Readiness 
Levels. These seem to be very important in the transition process. 
Although the management plans don't specify exactly what TRL the 
program enters into the ACTD our judgment was based on how much of 
the program was partial or all Commercial Off The Shelf (COTS). By 
introducing a program that is lower on the TRL scale will most likely 
decrease its chances of having a successful transition. It is very important 
to obtain the highest maturity level when entering a program into the 
ACTD process. The more items in a program that are COTS the better 
chance of a transition will occur. A future study of program TRLs is 
recommended due to the fact that we found it to be statistically significant 
in our study. This would entail finding the TRL for each and every COTS 
item used in a program to determine the overall Technology Readiness 
Level. 


Adequate and coordinated funding is a must for any program. As we 
stated earlier the funding must match the schedule in order for the program 
to have a chance to transition. Having the budget in place will increase 
the chances of a transition. This is not anything new. We concentrated on 
whether the budget actually matched the schedule. This seemed to have 
an effect on the outcome of the program. If a program had a 3 year budget 
set in place and it was scheduled as a 4 year process then this will most 
likely prevent the program from transitioning. By requiring the lead 
service to plan a budget around the schedule would help the process. This 
would require the program to initiate a request for Planning, Program, and 
Budgeting system funds associated with the expected Fiscal Year of 
transition. Correct planning will prevent the program from loosing 
momentum or being stalled to the point of no longer being of value 2 years 
later when the budget is approved. A closer look at the budget compared 
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to the schedule will be in order. Our study shows that this is another 
statistically significant issue on whether a program transitions or not. 


a A realistic risk assessment must be included within the management plan. 
Including a lessened estimate of programmatic risk may make the 
immediate plan more readily acceptable but does not institute a stable 
framework to further develop the program. In addition to realism, the risk 
assessment must also be detailed enough to delineate mitigation efforts to 
offset known risks. Mitigation efforts must be at a detail to demonstrate a 
competent understanding that the risks facing a program are 
acknowledged and understood. This may make the initial plan harder to 
accept, though it portrays a realistic understanding of the program being 
developed. Risk assessment is important to the transition process. Even 
though a subjective analysis was used to categorize this particular variable 
having a solid risk mitigation plan in place will help any program. By 
making the lead sponsor put the risk level in the management plan will 
force the teams to understand where the program stands and how difficult 
the transition can be. The programs that we coded as high risk normally 
did not address risk in any form within the management plan. A 
recommendation to specifically address the actual risk the program 
involves will possibly enhance the transition of ACTDs. 


We also made another observation during this study, which we mention here for 
completeness. It seems that the military in general lives off a Standard Operating 
Procedure (SOP) in everything we do. Looking through multiple ACTD management 
plans the lack of standardization was very obvious. If a standardized template along with 
what information is required was put into place it would eliminate submission of a sub- 
standard plan. Our observations in general were the management plans that incorporated 
every suggestion that is listed on the ACTD website transitioned. Even though many that 
did not follow these suggestions also transitioned but over all the plans that transitioned 
addressed a majority of the ACTD suggestions. On average the programs that 


transitioned had a page count of 29.3 pages where the non-transition programs had 14.3 


pages. 
B. CONCLUSIONS 


The problem with any acquisition program is the time acquired for 
implementation. By changing an ACTD program today the results will not surface for a 


minimum of two years. Even though we would like to be proactive in improving the 
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process, it is very hard when the time line for the results is in the distant future. The best 
action to take is make the change and wait. The dilemma is by only changing one thing 
at a time will take forever to get the process efficient enough to rely on. On the other 
hand if you change to many things at once you might not be able to determine the true 
reason for the change. One change may be less relevant than another. If you continue to 
improve on the lesser of the changes this may actually give a negative result in the long 
run. Therefore improving the ACTD process may take time. It seems that the best 
course of action would be patience and change one recommendation at a time and see 
what the results bring. This method would narrow down the most important problems 
that are keeping ACTDs from transitioning into the normal acquisition process. By 
changing multiple issues at the same time will prevent the largest problem from 


surfacing. This could cause a much larger delay in correcting the more important issues. 


In May of 2003 new acquisition instructions were approved for implementation 
following the Secretary of Defenses August 2002 cancellation. Associated with these 
processes are mechanisms that are designed to foster efficiency and innovation in 
conjunction with future Evolutionary Acquisition Strategies. Evolutionary acquisition 
strategies are the preferred approach to satisfying operational needs. The two 
mechanisms that have been identified include incremental development and spiral 
development. Under incremental development the end-state requirement is known and 
will be met over time through several system increments, or configurations. Under Spiral 
development the desired capabilities are identified but the — end-state 
capabilities/requirements are not specifically known at program initiation. Spiral 
development is an iterative process that links users to developers through an approach of 
continuous development and deployment of both software and hardware. The end-state 
capabilities/requirements for the future increments are dependent upon technology 
maturation and user feedback from the initial increments. Of these two mechanisms, 


spiral development shall be the preferred process (Department of Defense, 2003). 


We have examined 11 variables to determine their effect on the outcome of an 
ACTD program. Of the 11 variables investigated, three proved to have a measurable 
relationship to the ultimate determination of an ACTD’s success. Budget Matched to 


a 


Schedule, Technology Maturity, and Risk Assessment were the three variables that 
demonstrated a correlation to an ACTD’s transition. It is our conclusion that when 
supplied with management plans for ACTD programs, decision authorities should further 


scrutinize these areas as they have an impact on the life of the program. 
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